"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part

JANOG50での登壇資料です

Transcript

1. "Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part Shu Mukai LINE Service Network 1 Team 2022/07/13@JANOG50
2. Agenda • Data Center Network概要 • Multi-AZs対応前のBackbone Networkと課題 • Multi-AZs対応後のBackbone Network
3. l プロフィール l 名前：向井脩 (Shu Mukai) l 居住地：福岡県福岡市 l 職歴 l 2015年〜2020年：KDDI Backbone Network, AS2516の設計・構築・運⽤ ピアリング担当 l 2021年〜 : LINE ITSC Service Network 1 Team Inter-DC Backbone Network, AS38631, IP CLOSの設計・構築・運⽤ SONiC導⼊検討 ABOUT ME
4. Data Center Network概要
5. Data Center Network Legacy CLOS ref:https://www.janog.gr.jp/meeting/janog43/application/files/7915/4823/1858/janog43-line-kobayashi.pdf
6. Data Center Network External Spine Leaf Leaf ToR ToR Spine 0.0.0.0/0 Leaf Leaf ToR ToR Other DC routes Server Server Core DC Network 192.0.2.1/32 192.0.2.2/32 192.0.2.0/24 192.0.2.2/32 192.0.2.1/32 NW-POP Router Backbone Router
7. Data Center Network External Spine Leaf Leaf ToR ToR Spine Leaf Leaf ToR ToR Server Server Internet Traffic Inter-DC Traffic Core DC Network Internet Traffic, Inter-DC Traffic, East-West Traffic 全て、各種LINEサービスの正常稼働に必要不可⽋ East-West Traffic
8. Multi-AZs対応前の Backbone Networkと課題
9. The Internet Backbone network before multi-AZs CLOS CLOS Tokyo DC-1 Legacy Osaka DC Singapore DC Mainly serve JP/TW user DR site of Tokyo The Internet The Internet Mainly serve TH user
10. The Internet Routing for Internet Traffic CLOS CLOS Legacy The Internet The Internet Default route Default route Full routes Full routes Default route Full Routes Tokyo DC-1 Osaka DC Singapore DC
11. The Internet Routing for Internet Traffic CLOS CLOS Legacy The Internet The Internet Osaka Routes Tokyo Routes Tokyo Routes Osaka Routes SG Routes SG Routes Tokyo DC-1 Osaka DC Singapore DC
12. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The Internet The Internet Osaka routes Tokyo routes SG routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
13. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The Internet The Internet Other DC routes Other DC routes Other DC routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
14. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow Inter-DC Traffic flow The Internet The Internet SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
15. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow Inter-DC Traffic flow The Internet The Internet Tokyo DC-1のネットワーク機器で障害が発⽣すると、 JP/TWで、LINE各種サービスに⼤きな影響が発⽣する SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
16. Multi-AZs対応後の Backbone Network
17. l Tokyo Region内に複数のAZを構築 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは、 影響を受けない構成 Design policy for multi-AZs network
18. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC
19. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 関東地⽅に新たに2つのDCを構築し、3つのDCを三⾓形で接続 DC間を接続する専⽤線は、複数社から異経路で調達 l 関東地⽅の3つのDCをAZ1〜AZ3と定義 Osaka DC Singapore DC
20. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC l Tokyo DC-1の⼀部サーバルームを、Legacy→CLOSへリノベーション NFVのmulti-AZs対応のためにCLOS構成が必要であるため。 ※Legacy構成のサーバールーム（non-AZサーバールーム）も引き続き運⽤中
21. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l NW-POP RouterをBackbone Routerに接続 l Backbone Networkで、Internet Traffic⽤VRFを新規に作成 l Internet Traffic⽤のRoutingを是正 Osaka DC Singapore DC
22. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes SG Routes SG Routes Osaka DC Singapore DC
23. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Full routes Default route Full routes Default route Default route Full Routes Osaka DC Singapore DC
24. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic l 平常時は、IGP COSTに従い東京NW-POPを利⽤ Osaka DC Singapore DC
25. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
26. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
27. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 東京NW-POP障害時は、⼤阪NW-POPを利⽤ VRF for Internet Traffic Osaka DC Singapore DC
28. Osaka NW-POPを利⽤ Tokyo Region内に新規 にNW-POPを構築 Pros Cons • 既存のネットワーク機器を流用で きる • 東日本ユーザ向けのlatencyの増加 がない • 東日本ユーザ向けのlatencyが増加 する • ネットワーク機器を新規に調達す る必要がある • 関東圏内での冗長では、NW-POP 冗長として不十分ではないか？ NW-POP Redundancy
29. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka routes AZ1 routes SG routes AZ3 routes AZ2 routes VRF for Inter-DC Traffic Osaka DC Singapore DC
30. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Other DC routes Other DC routes Other DC routes Other DC routes Other DC routes VRF for Inter-DC Traffic Osaka DC Singapore DC
31. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
32. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
33. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
34. l ⼤阪からインターネットへ流すと、東⽇本ユーザ向けのlatencyが増加 l オリジンが東⽇本にあるため、仕⽅がない l トランジットコスト増加 l ⼤阪でも、東京と同容量のトランジット確保が必要 →コミット値の上昇→トランジットコスト増加 l ピアリングで捌けるトラフィック量を少しでも増やしたい 課題
35. l 関東地⽅に新たに2つのDCを構築し、それぞれのDCをAZとして定義 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは影響を受けない Backbone Network l NW-POP TokyoのBackupとして、NW-POP Osakaを利⽤ Conclusion
36. THANK YOU