customers is driving storage needs that look exponential or cubic rather than linear Keeping ahead of this curve is the responsibility of our infrastructure team In the last year, our Kubernetes (k8s) fleet size doubled and we energized >10 petabytes storage service capacity
Kubernetes development lifecycle for storage services across multiple substrates • Leveraging local storage, cloud-native, and secure • Immutable containers, declarative manifests, and active reconciliation • From manifest check in, to production in less than 30 minutes
data loss • For example: Ceph, Cassandra, Bookkeeper, HDFS, HBase Applications with intensive read/write profiles • Large fast on-disk caches • Avoid cold restarts • Interactive analytic applications
Persistent Volume (PV) • Must be formatted and mounted first • Dynamic provisioning NOT supported (yet) Scheduler enhancements • Data gravity • Volume binding looks at Pod requirements • Multiple PVCs in a Pod
with local SSDs 2. Node VM setup script formats and mounts local SSDs to discovery directories for LV provisioner Specific to Google Kubernetes Engine environment https://cloud.google.com/kubernetes-engine/docs/concepts/local-ssd
and declare servers configurations 2. Nodeprep daemonset scans new servers 3. Performs volume operations for desired resources a. Partition, clean, and mount 4. Mounts or links resources to discovery directories for LV provisioner 5. Marks node with nodeprep complete label for Daemonset magic
2. Creates local PVs 3. Workload consumes and releases PV 4. Volume data cleaned, and PV deleted 5. Repeat Open source LV provisioner that runs in any Kubernetes cluster https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume
• Node prep automation, environment specific • Static local PV provisioner After environment is setup, local PVs are ready for consumption • Same PVC/PV interface as remote storage • Best with StatefulSets
and works with LV provisioner v2.1.0 • Higher performance by bypassing FS • Small objects stored in a database • Example: Ceph Luminous Bluestore/Bluefs metadata Dynamic provisioning with LVM • Improved local disk utilization • But performance penalty of shared disks Handle FS formatting and mounting in Kubernetes