Illustrated Guide To Kubernetes Networking

A short walk through of some ideas around container networking.

1. Google Cloud Platform An Illustrated Guide to Kubernetes Networking Tim Hockin <thockin@google.com> Senior Staff Software Engineer @thockin
2. Google Cloud Platform Layer 2: ethernet
3. Google Cloud Platform node-a node-c node-b node-d 192.168.1.1/16 11:22:33:44:55:01 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 switch
4. Google Cloud Platform node-a node-c node-b node-d L2 to: 192.168.1.3 from: 192.168.1.1 GET / 192.168.1.1/16 11:22:33:44:55:01 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04
5. Google Cloud Platform node-d node-b node-a node-c L2 to: <broadcast> from: 11:22:33:44:55:01 who has 192.168.1.3? to: 192.168.1.3 from: 192.168.1.1 GET / 192.168.1.1/16 11:22:33:44:55:01 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 “ARP request”
6. Google Cloud Platform node-a node-c node-b node-d L2 to: 192.168.1.3 from: 192.168.1.1 GET / 192.168.1.1/16 11:22:33:44:55:01 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 to: 11:22:33:44:55:01 from: 11:22:33:44:55:03 I have 192.168.1.3 “ARP response”
7. Google Cloud Platform node-a node-c node-b node-d L2 to: 192.168.1.3 via: 11:22:33:44:55:03 from: 192.168.1.1 GET / 192.168.1.1/16 11:22:33:44:55:01 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04
8. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 11:22:33:44:55:01 L2 with containers cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24
9. Google Cloud Platform node-a 192.168.1.1/16 11:22:33:44:55:01 node-c node-b node-d 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 ctr-1 10.0.1.2 aa:bb:cc:dd:e1:01 ctr-2 10.0.3.2 aa:bb:cc:dd:e3:02
10. Google Cloud Platform node-a 192.168.1.1/16 11:22:33:44:55:01 node-c node-b node-d 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 ctr-1 10.0.1.2 aa:bb:cc:dd:e1:01 ctr-2 10.0.3.2 aa:bb:cc:dd:e3:02 to: 10.0.3.2 from: 10.0.1.2 GET /
11. Google Cloud Platform node-a 192.168.1.1/16 11:22:33:44:55:01 node-c node-b node-d 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 ctr-1 10.0.1.2 aa:bb:cc:dd:e1:01 ctr-2 10.0.3.2 aa:bb:cc:dd:e3:02 to: 10.0.3.2 from: 10.0.1.2 GET / to: <broadcast> from: aa:bb:cc:dd:e1:01 who has 10.0.3.2? “ARP request”
12. Google Cloud Platform node-a 192.168.1.1/16 11:22:33:44:55:01 node-c node-b node-d 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 ctr-1 10.0.1.2 aa:bb:cc:dd:e1:01 ctr-2 10.0.3.2 aa:bb:cc:dd:e3:02 to: 10.0.3.2 from: 10.0.1.2 GET / to: aa:bb:cc:dd:e1:01 from: 11:22:33:44:55:03 I have 10.0.3.2 “proxy ARP response”
13. Google Cloud Platform node-a 192.168.1.1/16 11:22:33:44:55:01 node-c node-b node-d 192.168.1.2/16 01:23:45:67:89:02 192.168.1.3/16 11:22:33:44:55:03 192.168.1.4/16 01:23:45:67:89:04 L2 ctr-1 10.0.1.2 aa:bb:cc:dd:e1:01 ctr-2 10.0.3.2 aa:bb:cc:dd:e3:02 to: 10.0.3.2 via: 11:22:33:44:55:03 from: 10.0.1.2 GET /
14. Google Cloud Platform Layer 3 - IP
15. Google Cloud Platform node-a node-c node-b node-d 192.168.1.1/32 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 gateway
16. Google Cloud Platform node-a node-c node-b node-d 192.168.1.1/32 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 to: 192.168.1.3 from: 192.168.1.1 GET /
17. Google Cloud Platform node-a node-c node-b node-d 192.168.1.1/32 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 to: 192.168.1.3 from: 192.168.1.1 GET /
18. Google Cloud Platform node-a node-c node-b node-d 192.168.1.1/32 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 to: 192.168.1.3 from: 192.168.1.1 GET / routing decision, static or learned (e.g. BGP)
19. Google Cloud Platform node-a 192.168.1.1/32 node-c node-b node-d 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 ctr-1 10.0.1.2 ctr-2 10.0.3.2
20. Google Cloud Platform node-a 192.168.1.1/32 node-c node-b node-d 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 10.0.1.2 GET /
21. Google Cloud Platform node-a 192.168.1.1/32 node-c node-b node-d 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 10.0.1.2 GET /
22. Google Cloud Platform node-a 192.168.1.1/32 node-c node-b node-d 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 10.0.1.2 GET / routing decision, static or learned (e.g. BGP)
23. Google Cloud Platform node-a 192.168.1.1/32 node-c node-b node-d 192.168.1.2/32 192.168.1.3/32 192.168.1.4/32 L3 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 10.0.1.2 GET /
24. Google Cloud Platform Overlays Q: When should I use an overlay? A: When nothing else works, or when you have specific reasons to want it (e.g. the added value of management)
25. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 Overlay (e.g. flannel, weave) cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24 flannel0: 10.0.1.254/16
26. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24 flannel0: 10.0.1.254/16 to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
27. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24 flannel0: 10.0.1.254/16 to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
28. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24 flannel0: 10.0.1.254/16 to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
29. Google Cloud Platform node-a root netns eth0: 192.168.1.1/16 cbr0: 10.0.1.1/24 ctr-1 eth0: 10.0.1.2/24 ctr-2 eth0: 10.0.1.3/24 ctr-3 eth0: 10.0.1.4/24 flannel0: 10.0.1.254/16 to: 192.168.1.3 from: 192.168.1.1 encap: to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
30. Google Cloud Platform node-a 192.168.1.1/16 node-c node-b node-d 192.168.1.2/16 192.168.1.3/16 192.168.1.4/16 ctr-1 10.0.1.2 ctr-2 10.0.3.2 Overlay (e.g. flannel, weave)
31. Google Cloud Platform node-c root netns eth0: 192.168.1.3/16 cbr0: 10.0.3.1/24 ctr-4 eth0: 10.0.3.2/24 ctr-5 eth0: 10.0.3.3/24 ctr-6 eth0: 10.0.3.4/24 flannel0: 10.0.3.254/16 to: 192.168.1.3 from: 192.168.1.1 encap: to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
32. Google Cloud Platform node-c root netns eth0: 192.168.1.3/16 cbr0: 10.0.3.1/24 ctr-4 eth0: 10.0.3.2/24 ctr-5 eth0: 10.0.3.3/24 ctr-6 eth0: 10.0.3.4/24 flannel0: 10.0.3.254/16 to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
33. Google Cloud Platform node-c root netns eth0: 192.168.1.3/16 cbr0: 10.0.3.1/24 ctr-4 eth0: 10.0.3.2/24 ctr-5 eth0: 10.0.3.3/24 ctr-6 eth0: 10.0.3.4/24 flannel0: 10.0.3.254/16 to: 10.0.3.2 from: 10.0.1.2 GET / Overlay (e.g. flannel, weave)
34. Google Cloud Platform Overlays - the hard part
35. Google Cloud Platform node-a 192.168.1.1/16 node-c non-node node-d 192.168.1.2/16 192.168.1.3/16 192.168.1.4/16 ctr-1 10.0.1.2 ctr-2 10.0.3.2 Overlay (e.g. flannel, weave)
36. Google Cloud Platform node-a 192.168.1.1/16 node-c non-node node-d 192.168.1.2/16 192.168.1.3/16 192.168.1.4/16 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 192.168.1.2 GET / Overlay (e.g. flannel, weave)
37. Google Cloud Platform node-a 192.168.1.1/16 node-c non-node node-d 192.168.1.2/16 192.168.1.3/16 192.168.1.4/16 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 192.168.1.2 GET / Overlay (e.g. flannel, weave)
38. Google Cloud Platform node-a 192.168.1.1/16 node-c non-node node-d 192.168.1.2/16 192.168.1.3/16 192.168.1.4/16 ctr-1 10.0.1.2 ctr-2 10.0.3.2 to: 10.0.3.2 from: 192.168.1.2 GET / ?!?! Overlay (e.g. flannel, weave)
39. Google Cloud Platform We need a bridge between the physical and overlay networks...
40. Google Cloud Platform We need a bridge between the physical and overlay networks... • could: route to nodes • could: route to 1 or more bridge machines • could: run flannel on client machines
41. Google Cloud Platform We need a bridge between the physical and overlay networks... • could: route to nodes • could: route to 1 or more bridge machines • could: run flannel on client machines • see “When should I use an overlay?”