Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes and Networks - why is this so dang h...

Kubernetes and Networks - why is this so dang hard?

A brief look at models for integrating Kubernetes clusters into existing networks.

Tim Hockin

July 10, 2020
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Kubernetes clusters are made up of nodes • Machines -

    virtual or physical Those nodes exist on some network Pods run on those nodes Pods get IP addresses “Network model” describes how those pod IPs integrate with the larger network What does “network model” mean?
  2. NOTE: It’s not *required* that a cluster be a single

    IP range, but it’s common and makes the pictures easier
  3. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24

    Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 NOTE: Different Ranges
  4. NOTE: It’s not *required* that nodes have a predefined IP

    range, but it’s common and makes the pictures easier
  5. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24

    Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-b: 10.0.1.2 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2
  6. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24

    Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-b: 10.0.1.2 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2
  7. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24

    Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-b: 10.0.1.2 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Other: 10.128.1.1 ?
  8. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 ? ?
  9. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2
  10. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 NOTE: Different Ranges
  11. Each node owns an IP range from the larger network

    Everyone on the network knows how to deal with that (or the network deals with it for them)
  12. Good when: • IP space is available • Network is

    programmable / dynamic • Need high integration / performance • Kubernetes is a large part of your footprint
  13. Bad when: • IP fragmentation / scarcity • Hard-to-configure network

    infrastructure • Kubernetes is a small part of your footprint
  14. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2
  15. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 NOTE: Same Range
  16. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  17. Good when: • Don’t need integration • IP space is

    scarce / fragmented • Network is not programmable / dynamic
  18. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 gateway gateway gateway
  19. Ingress and egress traffic goes thru one or more abstract

    “gateways” (more on that later)
  20. You can re-use the Pod IPs (a major motivation for

    this model), but node IPs come from the larger network
  21. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.1 Pod

    range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 gateway gateway gateway NOTE: Same Range
  22. Another way to think of this: clusters have a private

    network for their pods; nodes have one leg in the main network and one leg in the cluster network
  23. Cluster A pods 10.0.0.0/16 Cluster B pods 10.0.0.0/16 “Main” network

    10.0.0.0/8 Cluster A Nodes Cluster B Nodes “Hole” 10.0.0.0/16 Other Any pod can reach the “main” network by masquerading as its node, but not vice-versa (except via a gateway)
  24. Good when: • Need some integration • IP space is

    scarce / fragmented • Network is not programmable / dynamic
  25. Bad when: • Need to debug connectivity • Need direct-to-endpoint

    communications • Need a lot of services exposed (especially non-HTTP) • Rely on client IPs for firewalls • Large number of nodes
  26. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  27. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  28. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  29. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  30. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  31. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  32. You can ingress L4 into an L7 proxy and forward

    from there (e.g. Ingress controllers)
  33. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  34. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  35. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  36. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  37. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  38. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  39. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2
  40. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 VIP VIP
  41. Network: 10.0.0.0/8 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: 10.0.1.0/24 Node2: 10.0.2.0/24

    Node1: IP: 10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.0.0.0/16 Node1: 10.1.1.0/24 Node2: 10.1.2.0/24 Node1: IP: 10.240.0.3 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Proxy Proxy
  42. Can either route to NodePort or directly to pod IPs

    (e.g. proxy has special config to “get onto the island”)
  43. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP:

    10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.2.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 gateway
  44. Network: 10.0.0.0/8 Cluster: 10.0.0.0/16 Other: 10.128.1.1 Cluster: 10.0.0.0/16 Node1: IP:

    10.240.0.1 Pod range: 10.0.1.0/24 Node2: IP: 10.240.0.2 Pod range: 10.0.2.0/24 Pod-a: 10.0.1.1 Pod-c: 10.0.2.1 Pod-d: 10.0.2.2 Pod-b: 10.0.1.2 Cluster: 10.1.0.0/16 Node1: IP: 10.240.0.3 Pod range: 10.1.1.0/24 Node2: IP: 10.240.0.4 Pod range: 10.1.1.0/24 Pod-a: 10.1.1.1 Pod-c: 10.1.2.1 Pod-d: 10.1.2.2 Pod-b: 10.1.1.2 gateway NOTE: Different Ranges
  45. Archipelago A pods 10.0.0.0/14 Archipelago B pods 10.0.0.0/14 “Main” network

    10.0.0.0/8 Cluster A Nodes Cluster C Nodes “Hole” 10.0.0.0/14 Other Cluster B Nodes Cluster D Nodes
  46. Good when: • Need high integration across clusters • Need

    some integration with non-kubernetes • IP space is scarce / fragmented • Network is not programmable / dynamic
  47. Bad when: • Need to debug connectivity • Need direct-to-endpoint

    communications • Need a lot of services exposed to non-k8s • Rely on client IPs for firewalls • Large number of nodes across all clusters