tenant network • LINE message service and related services running Exclusive Network for Services • Service with specific requirements running • Building specific network for each service * Excitingly simple multi-path OpenStack networking: LAG-less, L2-less, yet fully redundant https://www.slideshare.net/linecorp/excitingly-simple-multipath-openstack-networking-lagless- l2less-yet-fully-redundant Many fragment underlay networks Many works to design and build Management cost increases Messaging, Manga, Game, ... Financial, HealthCare ...
VM VM Packet VM Private Network B VM VM VM • Virtual machine connects to private network • Isolated from other private networks • Multiple networks for each tenant
Cons • Lose advances of full-L3 • Need additional protocol to achieve service chaining IPv6 Segment Routing (SRv6) Pros • IPv6 forwarding only on underlay • Support segregation and service chaining with Segment ID Cons • No information about DC use case • No network device support + SRv6 future Adopted SRv6 Multi-tenancy Network
Information for routing to SRv6 node(parent node). It must be unique whitin a SR domain • Function: Information to identify the action to be performed on the parent node Segment Routing Header (SRH) • IPv6 extension header • Including a Segment List, Segment Left points out current point of Segment List and so on Locator Function Function examples • H.Encaps(Encap): Encapsulation packet with IPv6 header and SRH • End.DX4(Decap): Remove IPv6 header and SRH from packet and then forward next hop • End.DT4(Decap): Remove IPv6 header and SRH from packet and then lookup routing table and forward (DT4 is implemented in Linux Kernel 5.11 but we donʼt support the kernel so we uses DX4 although DT4 is better) 128bit
VM VRF VRF Hypervisor Server VM VM VM VRF VRF SRv6 Header Packet • On Hypervisor server(HV), virtual network is achieved with Linux VRF • On IPv6 Network between HVs, virtual network is achieved with SRv6 overlay network
B VM vrfA SRv6 Header fc00:aaaa:bbbb:cccd::2 Packet fc00:aaaa:bbbb:cccc::1/128 fc00:aaaa:bbbb:cccd::1/128 lo lo fc00:aaaa:bbbb:cccc::2/128 vrfA fc00:aaaa:bbbb:cccd::2/128 vrfA Locator Function Packet • Hypervisor server has specific SID identifies a VRF of server • Locator: Identify hypervisor server (e.g. fc00:aaaa:bbbb:cccc::/64) • Function: Identify VRF(e.g. 2) • SRv6 Header with SID is added to head of packet from VM
all L3 switches(Cumulus) donʼt support SRv6 • All switches forwards SRv6 packet as just IPv6 packet Server ToR ToR ToR ToR ToR ToR Server Server Server Server Server Server Server Server Server Server IPv6 Packet IPv6 Packet IPv6 Packet VM
vrfA SRv6 Header fc00:aaaa:bbbb:cccc::2 Packet fc00:aaaa:bbbb:cccc::1/128 lo fc00:aaaa:bbbb:cccc::2/128 vrfA Locator Function Packet • Hypervisor server has SRv6 decapsulation rule • The rule checks function in SRv6 Header and forwards the packet without the header • Packet is forwarded to VM on Linux VRF fc00:aaaa:bbbb:cccd::2/128 action End.DT4 vrfA Packet
Virtual Machine(VM) with tap device • Each tap device behaves like default gateway for each VM Nova Compute However there is no route to each VM VM Linux Routing Cumulus Switch
default route to FRR on Hypervisor server • FRR gets default route and adds the route on Hypervisor server Now VMs get default route but VMʼs IP address is not reachable out side of Hypervisor server yet
route for each HV server • Neutron adds a route for each VM on VRF Packet with SRv6 header is reachable to each HV but VMʼs IP isnʼt reachable yet 73'#
each VRF which combines Locator and Function • Neutron adds End.DT4 rules Packet with SRv6 header can be decapsulated on each HV server but VMʼs packet isnʼt encapsulated yet 73'#
configurations are managed via Neutron API • Failure of controller layer may affect network management • We have to add controller algorithm for everything • Sometimes API response latency may be bottleneck • Difficult to connect to SRv6 support network device • Neutron needs to configure network device if necessary
IPv6 address route of each HV server • Neutron adds a route for each VM on VRF Packet with SRv6 header is reachable to each HV but VMʼs IP isnʼt reachable yet
SID for each VRF which combines Locator and Function • Neutron adds End.DT4 rules Packet with SRv6 header can be decapsulated on each HV server but VMʼs packet isnʼt encapsulated yet
BGP Server L3 Switch L3 L3 Server Server Server Server Server VM 10.0.0.1 10.0.0.2 Server L3 Switch L3 L3 Server Server Server VM 10.0.0.1 10.0.0.2 10.0.1.1 10.0.1.2 10.0.0.1 10.0.0.2 10.0.1.1 10.0.1.2 10.0.0.1 10.0.0.2 BGP to VM Without BGP to VM
address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM by Neutron • Use local AS number • Use metadata server IP(169.254.169.254) on HV
address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM
address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM 5. VMʼs FRR advertises the address to HVʼs FRR via BGP
address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM 5. VMʼs FRR advertises the address to HVʼs FRR via BGP 6. HVʼs FRR advertises the address to upper L3 switch
Server Cloud Router Cloud Router SRv6 Header Packet • SRv6 network between VMs with same private network • IPv4 network between VM and IDC network via Cloud Router • Secure network between VM and other network via Cloud Router IPv4 Packet Another location VPN
to VPN Gateway on other location 2. Cloud Router advertises Network1ʼs subnet address to VPN Gateway 3. Could Router forwards packets into IPsec tunnel
case • Architecture of SRv6 data networking • SRv6 SDN implementation • NFV implementation on SRv6 network • Future plan: SDN and BGP hybrid model Thank you