Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to design Data-Center Networking? LINE‘s SD...

How to design Data-Center Networking? LINE‘s SDN engineering case study

LINE Developers

December 03, 2021
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. How to design Data-Center Networking? LINE‘s SDN engineering case study

    Verda Network Development Team, LINE Corporation Hiroki Shirokura Internet Week 2021 1
  2. • Senior Software Engineer @ Private Cloud • Responsibility: SDN,

    Cloud Networking • Design / Implementation / Reliability • SRv6, BGP OSS Upstream Developer • FRRouting, ExaBGP, etc.. • https://github.com/slankdev/ • HN: slankdev I’m Hiroki Shirokura from LINE 3 I both Control-plane, Data-plane
  3. Agenda • About LINE Corporation and its infrastructure • Looking

    back LINE’s Software Defined Networking • Pain Point / Case Study / Knowledge 4
  4. Region-B Region-C Region-A Internet CLOS Aggregate NW Verda Verda Verda

    Dedicated Infra Dedicated Infra Dedicated Infra
  5. Region-1 Region-2 Region-3 STAGE MASTER STAGE Region-1 Region-1 Region-1 MASTER

    Region-1 Region-2 Region-1 Service A (real) Service B (real) Service C (real) Service A (stage/dev) Service B (stage/dev) Service C (stage/dev) Verda-Prod Verda-Dev For LINE’s Services For Feature QA For Feature Dev Total VMs 85,000+ (New 10k VMs / Half) Total PMs 30,000+ Total HVs 4,000+ Jul. 2021 https://superuser.openstack.org/arti cles/2020-superuser-award-nomine e-line/
  6. 3 SWEs for stable-services • system operator ◦ customer support

    ◦ maintenance • software developer • project manager
  7. Region-B Region-C Region-A Internet CLOS Aggregate NW Dedicated Infra Dedicated

    Infra Dedicated Infra For WHAT..? For WHAT..? For WHAT..?
  8. Region-B Region-C Region-A Internet CLOS Aggregate NW Dedicated Infra Dedicated

    Infra Dedicated Infra WHAT IS FOR..? WHAT IS FOR..? Fintech, HealthCare, etc..
  9. SHARED L3 NETWORK Batch Svr HTTP svr Computing Service (VM/PM)

    Managed Service MySQL Elastic Search K8s K8s K8s CP Elastic Search K8s K8s K8s Cluster K8s K8s K8s CP K8s K8s K8s Cluster Redis PM VM Heavy Workload External Service Notify System VM Kafka Service A Service B Service C Background: Virtual Private Cloud is needed CURRENT NETWORKING ISSUE-2 BIG SHARED ACL 15 Shared Big ACL ISSUE-1 NO IP-level isolation Between Each services AS-IS
  10. net-b1 net-a2 net-a1 net-c1 Batch Svr HTTP svr Computing Service

    (VM/PM) Managed Service MySQL Elastic Search K8s K8s K8s CP Elastic Search K8s K8s K8s Cluster K8s K8s K8s CP K8s K8s K8s Cluster Redis PM VM Heavy Workload Notify System VM Kafka Service A Service B Service C Background: Virtual Private Cloud is needed (1) Isolated private network (2) NFV services (L3-routing,VPN,ACL, etc..) 16 VPN Collaborator DC Server Server TO-BE
  11. What is SDN, Why we need SDN • What is

    Software Defined Networking ◦ Original Software Logic belongs to Company’s Business Logics for Network Control ◦ Well Known as: ▪ No many Logging-In to Network Equipment and updating configuration for Network Ops ▪ Be able to configure from Single Point to Many Network Equipments • Why we need Software Defined Networking ◦ Basically we love Commodity Logic instead of Original one ◦ Manything can’t be achieved with ONLY Commodity (ex: Automating EVPN, Its Configuration) ◦ It’s Difficult to make the Logic to fit for many cases ▪ Let’s device actual logic, But let’s unify the interface,database,etc.... ▪ That is the Sense and Approach of SDN 19 Without SDN With SDN
  12. SDN Architecture Variants • Type-1: Almost Dataplane Configuration is done

    by SDN ◦ SDN agents execute “ip route add xxx” to own network-system ◦ Can do anything, but high development cost • Type-2: Almost Controlplane(routing-proto) Configuration is done by SDN ◦ SDN agents execute “vtysh -c ‘router bgp 1 vrf vrf1’ -c ‘bgp router-id 1.1.1.1’” ◦ Some constraint exist, but low development cost ▪ Can use existing technology’s strong point ▪ ex: health check, maintenance technique, etc.. • Practice: Prioritize “Type-2 -> Type-1” ◦ For newer technology (like a srv6) will be used as Type-1 ◦ Few month/year later, it should be moved as Type-2 in some cases (*)These are defined for only this presentation 20
  13. Gen-1,2,3 SRv6 Overlay Network Design • Gen1: https://www.janog.gr.jp/meeting/janog44/program/srv6/ • Gen2,3:

    Overlay Network Terminator (Baremetal → vm) ◦ Maintenance of virtual router cluster can be controlled by SDN ◦ Lower physical equipment per each environment • Issues ◦ HealthCheck & Failover feature development cost and its flexiblity ◦ -> Type-1 development cost... 21
  14. SDN Architecture Variants • Type-1: Almost Dataplane Configuration is done

    by SDN ◦ SDN agents execute “ip route add xxx” to own network-system ◦ Can do anything, but high development cost • Type-2: Almost Controlplane(routing-proto) Configuration is done by SDN ◦ SDN agents execute “vtysh -c ‘router bgp 1 vrf vrf1’ -c ‘bgp router-id 1.1.1.1’” ◦ Some constraint exist, but low development cost ▪ Can use existing technology’s strong point ▪ ex: health check, maintenance technique, etc.. • Practice: Prioritize “Type-2 -> Type-1” ◦ For newer technology (like a srv6) will be used as Type-1 ◦ Few month/year later, it should be moved as Type-2 in some cases (*)These are defined for only this presentation 22
  15. draft-ietf-bess-srv6-services: SRv6 BGP based Overlay Services • Additional Sub-Type of

    Prefix SID Path Attribute ◦ [new] Type-5: L3VPN Service SID ◦ [new] Type-6: L2VPN Service SID ◦ Extension of IPVPN(RFC4364), EVPN(RFC7432) to support VPN with SRv6 in addition MPLS 23 PE1 VRF1 RD1:1 Export-RT 1 10.1.0.0/24 BGP BGP UPDATE type: BGP_UPDATE attrs: - MP_REARCH_NLRI(1:1:10.1.0.0/24,label=33) - ECOMMUNITY(Type=RouteTarget, val=1) BGP MPLS L3VPN BGP SRv6 L3VPN PE1 VRF1 RD1:1 Export-RT 1 10.1.0.0/24 BGP BGP UPDATE type: BGP_UPDATE attrs: - MP_REARCH_NLRI(1:1:10.1.0.0/24,label=3) - ECOMMUNITY(Type=RouteTarget, val=1) - PREFIX_SID(1::1) label=33 act=vrf1 SID=1::1 End.DT4(vrf1)
  16. SRv6 Domain Type-1 :: IPv6 Routing Proto + SDN Controller

    24 R1 (1::/64) R2 (2::/64) VRF1 VRF Def VRF1 VRF Def eth eth eth eth 10.1.0.0/24 10.2.0.0/24 > ip route add 10.2.0.0/24 \ encap seg6 mode encap \ segs 2::1 dev eth0 \ vrf vrf1 > ip route add 1::1 \ encap seg6local \ action End.DT4 \ vrftable 1 \ dev eth0 SDN Agent SDN Agent VRF2 VRF2 eth eth 10.2.0.0/24 10.2.0.0/24 > ip route add 10.1.0.0/24 encap seg6 mode encap \ segs 1::1 dev eth0 vrf vrf1 > ip route add 2::1 encap seg6local action End.DT4 \ vrftable 1 dev eth0 SDN Controller nodes: - { name: R1, locator: 1::/64 } - { name: R2, locator: 2::/64 } networks: - tenantID: 1 prefix: 10.1.0.0/24 sid: 1::1 - tenantID: 1 prefix: 10.2.0.0/24 sid: 2::1 - tenantID: 2 prefix: 10.1.0.0/24 sid: 1::2 - tenantID: 2 prefix: 10.2.0.0/24 sid: 2::2
  17. Type-2 :: All Routing Proto (BGP-SRv6-L3VPN) 25 SRv6 Domain R1

    (1::/64) R2 (2::/64) VRF1 RD1:1 Import-RT 1 Export-RT 1 VRF Def VRF1 RD2:1 Import-RT 1 Export-RT 1 VRF Def type: BGP_UPDATE attrs: - TYPE: MP_REARCH_NLRI PREFIX: 10.2.0.0/24 - TYPE: PREFIX_SID SUB_TYPE: 5(L3VPN) SID: 2::1 - TYPE: ECOMMUNITY SUB_TYPE: RouteTarget VALUE: 1 eth eth eth type: BGP_UPDATE attrs: - TYPE: MP_REARCH_NLRI PREFIX: 10.2.0.0/24 - TYPE: PREFIX_SID SUB_TYPE: 5(L3VPN) SID: 2::1 - TYPE: ECOMMUNITY SUB_TYPE: RouteTarget VALUE: 1 eth 10.1.0.0/24 10.2.0.0/24 > ip route add 10.2.0.0/24 \ encap seg6 mode encap \ segs 2::1 dev eth0 \ vrf vrf1 > ip route add 1::1 \ encap seg6local \ action End.DT4 \ vrftable 1 \ dev eth0 BGP AS1 BGP AS2 VRF2 RD1:2 Import-RT 2 Export-RT 2 VRF2 RD2:2 Import-RT 2 Export-RT 2 eth eth 10.2.0.0/24 10.2.0.0/24 > ip route add 10.1.0.0/24 encap seg6 mode encap \ segs 1::1 dev eth0 vrf vrf1 > ip route add 2::1 encap seg6local action End.DT4 \ vrftable 1 dev eth0 BGP UPDATE type: BGP_UPDATE attrs: - TYPE: MP_REARCH_NLRI PREFIX: 10.1.0.0/24 - TYPE: PREFIX_SID SUB_TYPE: 5(L3VPN) SID: 1::1 - TYPE: ECOMMUNITY SUB_TYPE: RouteTarget VALUE: 1 type: BGP_UPDATE attrs: - TYPE: MP_REARCH_NLRI PREFIX: 10.2.0.0/24 - TYPE: PREFIX_SID SUB_TYPE: 5(L3VPN) SID: 2::1 - TYPE: ECOMMUNITY SUB_TYPE: RouteTarget VALUE: 1 BGP UPDATE
  18. SDN Controller can everything, but it should keep simple Current

    SRv6 multi tenant network SDN mechanism is complicated with our special SDN controller. SDN has strong configurability, i.e. It can know everything in the network. But when it has something wrong, All world will be gone… Gen-4 SRv6 Overlay Network Design BGP VPNv4 SRv6 for SRv6 Multi-tenant Networking ref: https://speakerdeck.com/line_developers/srv6-bgp-control-plane-for-lines-dcn eBGP C-plane Neutron C-Plane eBGP C-Plane Neutron C-Plane Underlay Network Overlay Network for Service B Overlay Network for Service A ipv6 ucast sr agnet srgw agnet Underlay Network Overlay Network for Service B Overlay Network for Service A ipv6 ucast vpnv4 ucast lightweight agent We want to replace C-plane for SRv6 m-t nw with BGP VPNv4 is really stable architecture because this is standard specification. Our future SDN controller only configures Routing software. then FRRouting will work to construct SRv6 overlay
  19. SDN Controller can everything, but it should keep simple Current

    SRv6 multi tenant network SDN mechanism is complicated with our special SDN controller. SDN has strong configurability, i.e. It can know everything in the network. But when it has something wrong, All world will be gone… Gen-4 SRv6 Overlay Network Design BGP VPNv4 SRv6 for SRv6 Multi-tenant Networking ref: https://speakerdeck.com/line_developers/srv6-bgp-control-plane-for-lines-dcn eBGP C-plane Neutron C-Plane eBGP C-Plane Neutron C-Plane Underlay Network Overlay Network for Service B Overlay Network for Service A ipv6 ucast sr agnet srgw agnet Underlay Network Overlay Network for Service B Overlay Network for Service A ipv6 ucast vpnv4 ucast lightweight agent We want to replace C-plane for SRv6 m-t nw with BGP VPNv4 is really stable architecture because this is standard specification. Our future SDN controller only configures Routing software. then FRRouting will work to construct SRv6 overlay
  20. SDN Architecturing Knowledge(1) Design Software Automation Aware Network • Using

    Commodity Protocol to get simplicity for SDN Logic ◦ No inline healthcheck mechanism by SDN Logic ◦ No inline failover mechanism by SDN Logic ◦ In our case, The commodity specification is already exist ▪ VPNv4 with SRv6 backend ▪ Of course upstreaming cost was really high • Another good points: ◦ Recruitment, On-boarding, Reusability • But if there is no Commodity, we need to consider how to ◦ Make commodity? or Wait for commodity? or Type-1? 28
  21. SDN System Architecture Design Knowledge(2) NAT dplane performance issue and

    its kernel panic • About Distributed NAT routing architecture: linedevday/2020/2076 , gihyo/line2021/0002 • Background ◦ Increasing users after 1st release ◦ There were 6 Linux servers as NAT dplane ▪ They are working as act/act, No session state sync ▪ 8vCPU/8GB-RAM x6 = 48vCPU ▪ RPS/RSS are disabled → Only 6vCPU are working 30 Internnet Internnet Immediately after release Increased users NAT Dplane Client core Internnet
  22. SDN System Architecture Design Knowledge(2) NAT dplane performance issue and

    its kernel panic • We enable RPS to use all cores • Few days later… weird kernel panics are occured in some servers • Few weeks later… All dplane servers are downed one by one, due to the same issue… ◦ There are some 秘孔 to make the server downed... 31 Internnet RPS enabled core Internnet Internnet Increased users
  23. SDN System Architecture Design Knowledge(2) NAT dplane performance issue and

    its kernel panic • We enable RPS to use all cores • Few days later… weird kernel panics are occured in some servers • Few weeks later… All dplane servers are downed one by one, due to the same issue… ◦ There are some 秘孔 to make the server downed... 32 Internnet RPS enabled core Internnet Internnet Increased users Kernel Panic! It was HELL...
  24. SDN System Architecture Design Knowledge(2) NAT dplane performance issue and

    its kernel panic • Then, we disalbed RPS again • And we scaled out dplane nodes x3 (6 servers → 18 servers) • Lesson learned ◦ (1) If your environment isn’t Majority case, be careful for tuning (LWT-BPF, etc..) ◦ (2) Scale out is right ◦ (3) Almost user work-loads were HTTPs/HTTP, It was easy to maintain ◦ (4) Operation Rehearsal ◦ (5) Performance lab 33 Scaled out Internnet Increased users Internnet core Internnet
  25. It's ALWAYS been My Turn ? • Do nothing, but

    necessary route are disappear from VRF…? ◦ Hey Software Developer! What is that…!? ◦ Many system (sys-a → sys-b → sys-c → sys-d) ▪ sys-a is developed by us ▪ sys-b is developed by us ▪ sys-c is developed by us ▪ sys-d … ah... • Approach practice: Make it visible what is occured at there 35 $ kubectl describe routingendpoint service1-vks-gateway-endpoint3-27ae0f1277 | grep -A 1000 "^Events:" Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal BGPPeerEstablish 6m39s routingendpoint-controller Succeed to establish a BGP peer hostname=XXXXX asn=65001 Normal ExternalApiCallOpenStack 6m39s routingendpoint-controller Call PUT /v2.0/ports/cabb8c57-c6f2-4f9b-baba-865b1a75d08e $ kubectl get event LAST SEEN REASON OBJECT MESSAGE 5m33s BGPPeerEstablish routingendpoint/service1-vks-gateway-endpoint1-deea61c0c5 Succeed to establish a BGP p... 5m34s ExternalApiCallOpenStack routingendpoint/service1-vks-gateway-endpoint1-deea61c0c5 Call PUT /v2.0/ports/ce224ed... 5m32s BGPPeerEstablish routingendpoint/service1-vks-gateway-endpoint2-5db7658f19 Succeed to establish a BGP p... 5m33s ExternalApiCallOpenStack routingendpoint/service1-vks-gateway-endpoint2-5db7658f19 Call PUT /v2.0/ports/ebcd654... 5m32s BGPPeerEstablish routingendpoint/service1-vks-gateway-endpoint3-27ae0f1277 Succeed to establish a BGP p... 5m32s ExternalApiCallOpenStack routingendpoint/service1-vks-gateway-endpoint3-27ae0f1277 Call PUT /v2.0/ports/cabb8c5...
  26. Develop Unify Platform for next development to Make development easier,

    faster and stabler • Develop The System for the system • ex: Restructure current Internet Gateway service with KloudNFV 36 Kubernetes Kubebuilder, Controller-runtime (Custom Resource Feature) SDN App … SDN Controller SDN App SDN Controller SDN App SDN Controller NATaaS Infra NATaaS Base Framework SDN App SDN Controller Kubernetes Kubebuilder, Controller-runtime (Custom Resource Feature) SDN App … SDN Controller SDN App SDN Controller SDN App SDN Controller SDN App SDN Controller Technical debt Need to Develop Need to Develop
  27. Summary • Many Infrastructure Challenges at LINE ◦ Large scale

    private cloud ◦ Fintech/HealthCare support ◦ Many Original systems • Automation/SDN aware system/network/team design ◦ Use existing control plane if we can ◦ Upstream control plane if we can ◦ Scale out is right ◦ System for the system • Q: Software Engineer do it? Network Engineer do it? • A: Both senses are needed ◦ What is critical? What is pain point? by architectural level ◦ Act-Stb, Act-Act, 2N, N+1, Blast-radius, Extensibility, Scalability 39
  28. LINE Corporation IT Service Center Verda Network System … Network

    Dev Team Platform Dev Team SRE Team UIE Team QA Team … Service Network Team … • Netdev Team’s Responsibility are Overall SDN Design and Network Function Service in our Private Cloud • Load Balancer, Internet Gateway, VPC mechanism, etc.. Organization in Charge of Infrastructure Construction/Operation/Development @ LINE