cloud and datacenter management • though I’m a Linux guy :/ • Previous: VMware, Baidu • Feature maintainer of Kubernetes • HyperCrew: https://hyper.sh • Publications: Docker & Kubernetes Under the Hood • PhD candidate @ZJU: Large-scale cluster management and scheduling
containers started in sequence before the pod's normal containers are started. • Share volumes, perform network operations, and perform computation prior to the app containers.
The atomic scheduling unit • The process group in container cloud • Do right things • without modifying your container image • Kubernetes = Spring Framework • Pod = IoC Pod log app infra container volume init container
wraps and encapsulates several app containers into a logical group • Hypervisor container based runtime • hypervisor serves as a natural boundary of Pod
create sandbox Foo --> create container C --> start container C • stop container C --> remove container C --> delete sandbox Foo • Sandbox • Normally: the infra container • HyperContainer: hypervisor • with HyperKernel • a HyperStart process as PID 1 • setup mnt namespace, launch apps from the images etc
object • each tenant (created by Keystone) has its own Network • Network mapping to Neutron “net” • a Network Controller is responsible to manage Network lifecycle
can communicate with all other containers without NAT • Node reach container • all nodes can communicate with all containers (and vice-versa) without NAT • IP addressing • Pod in cluster can be addressed by its IP
one or more Namespaces • Pods belonging to the same Network can reach each other directly through IP • a Pod’s network mapping to Neutron “port” • kubelet is responsible for Pod network setup • let’s see how kubelet works
HandlePods {Add, Update, Remove, Delete, …} NodeStatus Network Status status Manager PLEG SyncLoop Pod Update Worker (e.g.ADD) • generale Pod status • check volume status (talk later) • call runtime to start containers • set up Pod network (see next slide) volume Manager PodUpdate image Manager
• Endpoint Pods and Nodes with iptables rules are isolated into different networks • Hypernetes uses a built-in HAproxy as the Service portal • to handle all Service instances within same namespace • the same OnServiceUpdate and OnEndpointsUpdate workflow • ExternalProvider • a OpenStack LB will be created as Service • e.g. curl 58.215.33.98:8078
mountPath mountPath attach mount Volume Manager desired World reconcile • Get mountedVolume from actualStateOfWorld • Unmount volumes in mountedVolume but not in desiredStateOfWorld • AttachVolume() if vol in desiredStateOfWorld and not attached • MountVolume() if vol in desiredStateOfWorld and not in mountedVolume • Verify devices that should be detached/unmounted are detached/unmounted • Tips: 1. -v host:path 2. attach VS mount 3. Totally independent from container management
Linux container: 1. full OpenStack cluster 2. query Nova to find node 3. attach Cinder volume to host path 4. bind mount host path to Pod containers • HyperContainer: • directly attach block devices to Pod • thanks to the hypervisor based Pod boundary • eliminates extra time to query Nova Host vol Enhanced Cinder volume plugin Pod Pod mountPath mountPath attach vol desired World reconcile Volume Manager