Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"Change" our private cloud infrastructures from...

"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part

JANOG50での登壇資料です

LINE Developers

July 13, 2022
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. "Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone

    Network part Shu Mukai LINE Service Network 1 Team 2022/07/13@JANOG50
  2. l プロフィール l 名前:向井脩 (Shu Mukai) l 居住地:福岡県福岡市 l 職歴

    l 2015年〜2020年:KDDI Backbone Network, AS2516の設計・構築・運⽤ ピアリング担当 l 2021年〜 : LINE ITSC Service Network 1 Team Inter-DC Backbone Network, AS38631, IP CLOSの設計・構築・運⽤ SONiC導⼊検討 ABOUT ME
  3. Data Center Network External Spine Leaf Leaf ToR ToR Spine

    0.0.0.0/0 Leaf Leaf ToR ToR Other DC routes Server Server Core DC Network 192.0.2.1/32 192.0.2.2/32 192.0.2.0/24 192.0.2.2/32 192.0.2.1/32 NW-POP Router Backbone Router
  4. Data Center Network External Spine Leaf Leaf ToR ToR Spine

    Leaf Leaf ToR ToR Server Server Internet Traffic Inter-DC Traffic Core DC Network Internet Traffic, Inter-DC Traffic, East-West Traffic 全て、各種LINEサービスの正常稼働に必要不可⽋ East-West Traffic
  5. The Internet Backbone network before multi-AZs CLOS CLOS Tokyo DC-1

    Legacy Osaka DC Singapore DC Mainly serve JP/TW user DR site of Tokyo The Internet The Internet Mainly serve TH user
  6. The Internet Routing for Internet Traffic CLOS CLOS Legacy The

    Internet The Internet Default route Default route Full routes Full routes Default route Full Routes Tokyo DC-1 Osaka DC Singapore DC
  7. The Internet Routing for Internet Traffic CLOS CLOS Legacy The

    Internet The Internet Osaka Routes Tokyo Routes Tokyo Routes Osaka Routes SG Routes SG Routes Tokyo DC-1 Osaka DC Singapore DC
  8. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The

    Internet The Internet Osaka routes Tokyo routes SG routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  9. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The

    Internet The Internet Other DC routes Other DC routes Other DC routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  10. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow

    Inter-DC Traffic flow The Internet The Internet SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  11. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow

    Inter-DC Traffic flow The Internet The Internet Tokyo DC-1のネットワーク機器で障害が発⽣すると、 JP/TWで、LINE各種サービスに⼤きな影響が発⽣する SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  12. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC
  13. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 関東地⽅に新たに2つのDCを構築し、3つのDCを三⾓形で接続 DC間を接続する専⽤線は、複数社から異経路で調達 l 関東地⽅の3つのDCをAZ1〜AZ3と定義 Osaka DC Singapore DC
  14. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC l Tokyo DC-1の⼀部サーバルームを、Legacy→CLOSへリノベーション NFVのmulti-AZs対応のためにCLOS構成が必要であるため。 ※Legacy構成のサーバールーム(non-AZサーバールーム)も引き続き運⽤中
  15. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l NW-POP RouterをBackbone Routerに接続 l Backbone Networkで、Internet Traffic⽤VRFを新規に作成 l Internet Traffic⽤のRoutingを是正 Osaka DC Singapore DC
  16. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes SG Routes SG Routes Osaka DC Singapore DC
  17. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Full routes Default route Full routes Default route Default route Full Routes Osaka DC Singapore DC
  18. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic l 平常時は、IGP COSTに従い東京NW-POPを利⽤ Osaka DC Singapore DC
  19. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
  20. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
  21. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 東京NW-POP障害時は、⼤阪NW-POPを利⽤ VRF for Internet Traffic Osaka DC Singapore DC
  22. Osaka NW-POPを利⽤ Tokyo Region内に新規 にNW-POPを構築 Pros Cons • 既存のネットワーク機器を流用で きる

    • 東日本ユーザ向けのlatencyの増加 がない • 東日本ユーザ向けのlatencyが増加 する • ネットワーク機器を新規に調達す る必要がある • 関東圏内での冗長では、NW-POP 冗長として不十分ではないか? NW-POP Redundancy
  23. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka routes AZ1 routes SG routes AZ3 routes AZ2 routes VRF for Inter-DC Traffic Osaka DC Singapore DC
  24. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Other DC routes Other DC routes Other DC routes Other DC routes Other DC routes VRF for Inter-DC Traffic Osaka DC Singapore DC
  25. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
  26. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
  27. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC