Share
Embed
"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part
JANOG50での登壇資料です
LINE Developers
July 13, 2022
Transcript
36 slides
-
"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone
Network part Shu Mukai LINE Service Network 1 Team 2022/07/13@JANOG50 -
l プロフィール l 名前:向井脩 (Shu Mukai) l 居住地:福岡県福岡市 l 職歴
l 2015年〜2020年:KDDI Backbone Network, AS2516の設計・構築・運⽤ ピアリング担当 l 2021年〜 : LINE ITSC Service Network 1 Team Inter-DC Backbone Network, AS38631, IP CLOSの設計・構築・運⽤ SONiC導⼊検討 ABOUT ME -
Data Center Network External Spine Leaf Leaf ToR ToR Spine
0.0.0.0/0 Leaf Leaf ToR ToR Other DC routes Server Server Core DC Network 192.0.2.1/32 192.0.2.2/32 192.0.2.0/24 192.0.2.2/32 192.0.2.1/32 NW-POP Router Backbone Router -
Data Center Network External Spine Leaf Leaf ToR ToR Spine
Leaf Leaf ToR ToR Server Server Internet Traffic Inter-DC Traffic Core DC Network Internet Traffic, Inter-DC Traffic, East-West Traffic 全て、各種LINEサービスの正常稼働に必要不可⽋ East-West Traffic -
The Internet Backbone network before multi-AZs CLOS CLOS Tokyo DC-1
Legacy Osaka DC Singapore DC Mainly serve JP/TW user DR site of Tokyo The Internet The Internet Mainly serve TH user -
The Internet Routing for Internet Traffic CLOS CLOS Legacy The
Internet The Internet Default route Default route Full routes Full routes Default route Full Routes Tokyo DC-1 Osaka DC Singapore DC -
The Internet Routing for Internet Traffic CLOS CLOS Legacy The
Internet The Internet Osaka Routes Tokyo Routes Tokyo Routes Osaka Routes SG Routes SG Routes Tokyo DC-1 Osaka DC Singapore DC -
The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The
Internet The Internet Osaka routes Tokyo routes SG routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC -
The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The
Internet The Internet Other DC routes Other DC routes Other DC routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC -
The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow
Inter-DC Traffic flow The Internet The Internet SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC -
The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow
Inter-DC Traffic flow The Internet The Internet Tokyo DC-1のネットワーク機器で障害が発⽣すると、 JP/TWで、LINE各種サービスに⼤きな影響が発⽣する SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC -
l Tokyo Region内に複数のAZを構築 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West
Trafficは、 影響を受けない構成 Design policy for multi-AZs network -
Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC -
Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 関東地⽅に新たに2つのDCを構築し、3つのDCを三⾓形で接続 DC間を接続する専⽤線は、複数社から異経路で調達 l 関東地⽅の3つのDCをAZ1〜AZ3と定義 Osaka DC Singapore DC -
Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC l Tokyo DC-1の⼀部サーバルームを、Legacy→CLOSへリノベーション NFVのmulti-AZs対応のためにCLOS構成が必要であるため。 ※Legacy構成のサーバールーム(non-AZサーバールーム)も引き続き運⽤中 -
Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l NW-POP RouterをBackbone Routerに接続 l Backbone Networkで、Internet Traffic⽤VRFを新規に作成 l Internet Traffic⽤のRoutingを是正 Osaka DC Singapore DC -
Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes SG Routes SG Routes Osaka DC Singapore DC -
Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Full routes Default route Full routes Default route Default route Full Routes Osaka DC Singapore DC -
Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic l 平常時は、IGP COSTに従い東京NW-POPを利⽤ Osaka DC Singapore DC -
Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC -
Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC -
Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 東京NW-POP障害時は、⼤阪NW-POPを利⽤ VRF for Internet Traffic Osaka DC Singapore DC -
Osaka NW-POPを利⽤ Tokyo Region内に新規 にNW-POPを構築 Pros Cons • 既存のネットワーク機器を流用で きる
• 東日本ユーザ向けのlatencyの増加 がない • 東日本ユーザ向けのlatencyが増加 する • ネットワーク機器を新規に調達す る必要がある • 関東圏内での冗長では、NW-POP 冗長として不十分ではないか? NW-POP Redundancy -
Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka routes AZ1 routes SG routes AZ3 routes AZ2 routes VRF for Inter-DC Traffic Osaka DC Singapore DC -
Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3
Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Other DC routes Other DC routes Other DC routes Other DC routes Other DC routes VRF for Inter-DC Traffic Osaka DC Singapore DC -
Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC -
Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC -
Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo
DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC -
l 関東地⽅に新たに2つのDCを構築し、それぞれのDCをAZとして定義 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは影響を受けない
Backbone Network l NW-POP TokyoのBackupとして、NW-POP Osakaを利⽤ Conclusion