Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS • Implement IPVS-based LB • Live Demo Agenda Today I would like to talk about Cloud Native Forum
flexible in how you can deploy it. You can deploy to cloud environments like Google Cloud, Microsoft Azure, and Amazon AWS. • You can deploy Kubernetes on bare metal using several popular operating systems like Ubuntu Linux, CentOS. On-premises
of network load-balancers (Services of type LoadBalancer) for bare metal(On-Premises) clusters. • Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. • Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.
ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer P.S. Does not offer an implementation of network load-balancers for bare metal clusters.
the Kubernetes API that stores a collection of API objects of a certain kind. ex: Pod. • A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. • Kubernetes provides two ways to add custom resources to your cluster: • CRDs • API Aggregation(custom apiserver)
feature called Custom Controllers. • It enables developers to extend and add new functionalities, replace existent ones (like replacing kube-proxy for instance). • And of course, automate administration tasks as if they were a native Kubernetes component.
set of application-specific custom controllers. • the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve the desired state. https://coreos.com/operators/ https://github.com/operator-framework/operator-sdk
load balancing, usually called Layer 4 LAN switching, as part of Linux kernel. • IPVS is incorporated into the LVS (Linux Virtual Server), where it runs on a host and acts as a load balancer in front of a cluster of real servers. • Same to IPTables, IPVS is built on top of Netfilter. • Support 3 load balancing mode: DNAT, DR(or DSR) and IP tunnel.
More load balancing algorithm • Round robin, source/destination hashing. • Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session
out of thin air, so you do have to give it pools of IP addresses that it can use. • Provides custom resources(IP and Pool). • Drive current state to desired state. • Allocate IP instance from Pool.
libipvs to create IPVS rules accordingly and syncs ipvs rules with Kubernetes Services periodically, to make sure IPVS status is consistent with the expectation.
and assign external IP, IPVS Node will do the following things: • Make sure a interface exists in the node, using bind-iface flag to specify. • Call go-netlink lib to bind Service external IP(VIP) addresses to the interface. • Call go-libipvs Create IPVS virtual servers for each Service external IP address respectively(Forward VIP to cluster IP). This rule wiil be used Round-robin algorithm.
type Service and assign external IP, IPVS proxier will do the following things: • Make sure a dummy interface exists in the nodes, defaults to kube-ipvs0. • Bind Service IP addresses(Cluster IP and External IP) to the dummy interface. • Create IPVS virtual servers for each Service IP address respectively.