Share
Embed
#25 “Swift: Delay is Simple and Effective for Congestion Control in the Datacenter”
ACM SIGCOM ’20, “Awarded best paper”
https://dl.acm.org/doi/abs/10.1145/3387514.3406591
cafenero_777
June 19, 2023
Transcript
25 slides
-
Research Paper Introduction #25 “Swift: Delay is Simple and E
ff ective for Congestion Control in the Datacenter” ௨ࢉ#78 @cafenero_777 2021/07/29 1 -
Agenda • ରจ • ֓ཁͱಡ͏ͱͨ͠ཧ༝ 1. INTRODUCTION 2. MOTIVATION 3.
SWIFT DESIGN & IMPLEMENTATION 4. TAKEAWAYS FROM PRODUCTION 5. EXPERIMENTAL RESULTS 6. RELATED WORK 7. CONCLUSION AND FUTURE DIRECTIONS 2 -
ରจ • Swift: Delay is Simple and E ff ective
for Congestion Control in the Datacenter • Gautam Kumar, Nandita Dukkipati, Keon Jang (MPI-SWS)∗, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, David Wetherall, and Amin Vahdat • Google LLC • ACM SIGCOM ’20, “Awarded best paper” • https://dl.acm.org/doi/abs/10.1145/3387514.3406591 3 -
֓ཁͱಡ͏ͱͨ͠ཧ༝ • ֓ཁ • Swift congestion controlͰԆݮ • <50us tail-latency@100Gbps
• DCTCPΑΓ10ഒԆɺincastʹͱͯڧ͍ • ಡ͏ͱͨ͠ཧ༝ͱײ • ON-RAMPͰҾ༻͞Ε͍ͯͨͨΊ • DC༻్Ͱ࡞ΒΕ͍ͯΔͨΊ • Best Paper@SIGCOM 2020 4 https://olympic-placard-generater.herokuapp.com -
1. Introduction • ੈେ”ࢄγεςϜ”࣌, buzz word: disaggregation X (Storage/Compute/Memory) •
TIMELY͔ΒSwiftਐԽ • E2Eͷਖ਼֬ͳԆଌఆΛ͏ɻSLOʹରԠ͍͢͠ɻϗετͱϑΝϒϦοΫͷԆ • NWͷػೳΛΘͳ͍ͷͰ࣍ੈεΠονͷػೳɾઃఆ͕͑Δ • ECNํࣜɿେنincastIOPS͕ଟ͍workloadʹෆ͖ɻ -> ϗετଆͷॲཧԆʹରॲͰ͖ͳ͍ɻi.e. DCTCP, HPCC • ύέοτεέδϡʔϦϯάํࣜɿεΠονͱ໌ࣔతʹεέδϡʔϦϯά -> ಋೖɾอक͕େมɺϚϧνςφϯτͩͱແཧɻɻi.e. pFabric • Swiftͷ݁Ռ • incastͳͲͷτϥϑΟοΫͰserver, switchͷΩϡʔΠϯάԆݮ, 100GbpsͰԆ, ΄΅zero packet loss • ༷ʑͳΞϓϦϨΠϠʔ͔ΒݟͯRPCੑೳྑ͍ • Ԇ৴߸ͱͯ͠γϯϓϧ͔ͭ༏ΕͨੑೳʢTIMELYΑΓγϯϓϧʣɻϗετଆʹରԠɻincastʹରԠ 6 -
2. MOTIVATION • Storage Workloads • HDD: O(10)ms -> Flash
O(100us) -> … • ࢄʢෳσόΠεΞΫηεʣ -> Tail-latencyͰ • Host Networking Stacks • kernelมߋίετߴ -> Snap (OS bypass & microkernel NW module)Ͱ࣮ • ϗετଆॲཧԆݕͤ͞Δ • Datacenter Switches • ੑೳɾػछόϥόϥͷͨΊɺ੍ޚΛ֤ػثͰӡ༻ࠔ • ྫɿECNόοϑΝαΠζௐ • ϗετଆγάφϧΛݩʹNWԆݮΛࢦ͢ 7 https://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf -
3. SWIFT DESIGN & IMPLEMENTATION (1/4) ઃܭࢦ 8 • Swiftͷࢦ
• େنDCNͰ༷ʑͳϫʔΫϩʔυΛଳҬɺԆ, near zero lossΛఏڙ • host (NIC/software stack)ͷ੍ޚͰE2E/NW FabricΛԆԽͤ͞Δ • CPUޮ͕ྑ͍͜ͱ • AIMD (Additive-Increase Multiplicative-Decrease )ΞϧΰϦζϜ • Ճࢉత૿ՃɾࢉతݮগΞϧΰϦζϜ • Ԇ࣌ؒΛNICؒ(fabric)ͱϗετؒʹ ? -
3. SWIFT DESIGN & IMPLEMENTATION (2/4) Ԇγάφϧͷཧ 9 QͰա࣌ؒ͢͝ (ૹ৴։࢝·Ͱͷ࣌ؒ)
NIC Qͪ࣌ؒ ߴෛՙͩͱԆܹ૿ ύέοτॲཧͯ͠ ackΛฦ͢·Ͱͷ࣌ؒ SW->hostͷ߹ܭ࣌ؒ ʢԆ͕ରশͰͳ͍Մೳੑ͋Γɻʣ ackύέοτॲཧͯ͠ ௨৴ྃ͢Δ·Ͱͷ࣌ؒ NWͷࢦඪ ϗετͷࢦඪ Γͷ௨৴ʢී௨ʣ੍ޚͰ͖ͳ͍ -> RemoteଆSwiftΛ੍͑ޚͰ͖Δ t4-t2ΛૹΔ host->SWͷ߹ܭ࣌ؒ ʢγϦΞϧԽɺΩϡʔΠϯάʣ t6-t1: E2EԆ (RTT) • E: ΤϯυϙΠϯτԆ: (t4-t2) + (t6-t5) • F: ϑΝϒϦοΫԆ: t1 - E F: E: -
3. SWIFT DESIGN & IMPLEMENTATION (3/4) cwndͷܾΊํ 10 • ඪԆͱ࣮ࡍͷԆͱͷࠩΛݩʹɺ
AIMDతʹʢՃࢉత૿ՃɾࢉతݮগʣcwndΛࢉग़ • ϑϩʔຖʹԆ • ϑΝϒϦοΫԆΛݩʹfcwndΛࢉग़ • ΤϯυϙΠϯτԆΛݩʹecwndΛࢉग़ • ༗ޮcwind := min (fcwnd, ecwnd) • ߴੑೳrwndͱࢥ͓͏ɻi.e. ૉͷcwindϝϞϦߟྀͷΈ • େنincastΛߟྀͯ͠cwnd1(packet)ҎԼͷ߹༗Γ • pacing_delay͏·͘࡞Δ • ͜ͷ࡞ઓͰtail-latency2ഒվળ ? มಈΠϝʔδ มಈΠϝʔδ -
3. SWIFT DESIGN & IMPLEMENTATION (4/4) Scaling the Target Delay
11 • λʔήοτԆʢfabricԆʣͷܾΊํ • τϙϩδґଘͷԆ૿Ճ • ܦ࿏ʹΑͬͯԆ͕ҧ͏ • ϑϩʔຖʹTTL(hop)ΛݟΔ -> λʔήοτԆʹม • ϑϩʔґଘͷԆ૿Ճ • ϑϩʔNʹԠͯ͡O(sqrt(N))ͰΩϡʔͷͪߦྻ૿Ճ • fair-shareঢ়ଶͰcwndϑϩʔʹൺྫ • Ԇ࣌ؒɿ t = base_target + #hops × ℏ + max(0, min( α fcwnd + β, fs_range)) τϙϩδґଘ ϑϩʔґଘ -
4. TAKEAWAYS FROM PRODUCTION (1/3) ProdͰͷଌఆ֓ཁ • 4͔͚ͯSwiftΛಋೖ • HDD/SSD
read/write (byte-intensive) • in-memory KVS (latency-intensive) • in-memory FS for BigQuery shu ff l e (IOPS-intensive) • ଌఆํ๏ • 1week, fl eet-wide, ෯͍ϫʔΫϩʔυɾنɾར༻ • εΠον౷ܭใʢlink-util, less-rate / portʣ • ϗετؒRTT (Swift, non-SwiftͷNIC timestamp) • ΞϓϦͷӨڹ • DCTCP૬ͱൺֱɻͨͩ͠ECNͰͳ͘GCNར༻ʢECNΛड৴ͨ͠ΒԆACKΛແޮԽ͢Δʣ 12 -
• Loss • Swiftͩͱ1-2ܻ͍ϩε • ΫϥελຖͩͱGCNมಈ͕େ͖͍ • Latency • ฏۉlatency
~= target delayΛୡ • ֎Ε: ߴQoS, heavy incast • E2EͰlantecy؇ • cwnd<1 • hostͰͷݕ 4. TAKEAWAYS FROM PRODUCTION (2/3) Loss & Latency 13 Fig6. Edge (ToR to host) links Fig9. ClusterEdge (ToR to host) links in the cluster Fig12. Cluster Swift/GCN RTT -
• wire-rateग़͍ͯΔͷʹϩε͕ͳ͍ -> ೋόάใࠂ͕དྷͨʂ • QoSຖʹݟΔʢnon-Swift QoSଆϩε͕͋Δͣʣ • εϧʔϓοτΛ٘ਜ਼ʹԆΛ࣮ݱͯ͠Δɺͱ͍͏ѱධʢʂʣ •
݁Ռͷѱ͍ผόʔδϣϯͱൺֱɻͰεϧʔϓοτվળͤͣɻΉ͠ΖRTT, loss૿Ճ • target-delayҾ্͖͛ɺcwnd<0ΛແޮԽͳͲ • SwiftͷCPU༻2.6%ఔʢ1.4%͕ackॲཧɺ0.31%͕࣌ؒଌఆॲཧʣ • ECNͷ͖͍͠ͷεΠονઃఆɾௐେมͩͬͨʢSwiftͷE2E latencyϗετଆ੍ޚͷͨΊ؆୯ʣ 4. TAKEAWAYS FROM PRODUCTION (3/3) Production Experience 14 -
5. EXPERIMENTAL RESULTS • ςετڥ • T1: 50Gbps host *
60 • T2: 100Gbps host * 500 15 T1: target delay@incast ~= RTTΛୡ֬ೝ 25usҎ্ͰthroughputΛୡ֬ೝ T2: RTT~25usఔͰSwiftͰthrouhgputୡΛ֬ೝ T1: େنincastͰSwiftͳΒ͑͏Δɻ ʢGCNͩͱ1000:1Ͱloss 5.1%, RTT-ave 8.6ഒʣ -
5. EXPERIMENTAL RESULTS • a 16 T1: workload(App)ຖʹՕॴ͕ҧ͏ T1: Swift-v0
(delay=RTT)Λ͍ɺincast (64kB & 60B)Λྲྀ͢ Swift (not v0)ThroughputͱRTTͷ྆ํྑ͍ IOPSܥ௨৴ fcwndେ͖͍ ecwndখ͍͞ʢhostԆʣ byteܥ௨৴ fcwndখ͍͞(fabricԆ) ecwndେ͖͍ -
5. EXPERIMENTAL RESULTS fl ow-fairness 17 T1: গͳ͍ϑϩʔͰެฏੑ֬ೝ T1: 50host*100
fl ow͕link্Ͱͨ͠߹ fl ow-based scalingʢʁʁʁʣ͕༗ޮ Jain fairness-index (J=0.91)͕ͱͯྑ͍ T1: path͕ҧ͏߹ Topology-base scaling༗ޮ intra-rack inter-rack -
6. RELATED WORK 18 TIMELY͔ΒSwiftͷਐԽ • fabric, hostͷ • RTTޯͰͳ͘E2EԆΛར༻
ʢਖ਼֬ͳtimestampΛackͰૹΔʣ • ෛՙɾτϙϩδʹԠͯ͡ඪԆΛscalingͤ͞Δ • େنincastରԠ -
7. CONCLUSION AND FUTURE DIRECTIONS • ੍ޚͷύϥυοΫε • ʢҰൠతʹ͑Δʣ੍ޚΛ࡞Γ͍ͨ •
ʢٯઆతʹʣεΠονɾϗετɺதԝཧɾΞϓϦຖͷௐ͕ඞཁʹͳΔ • GoogleͰͷؾ͖ • γϯϓϧͳγάφϧ͕ྑ͍ • NIC Timestamp, ϓϩτίϧελοΫվྑ • Swift: 100GbpsଳҬ@30us • طଘϓϩτίϧελοΫʹ࠶ར༻Ͱ͖ͦ͏ • <10usΛαϙʔτ͢Δʹߋʹ৽͍ٕ͠ज़͕ඞཁ 19 -
• ༷ʑͳΞϓϦͰ༗ޮ • in-memory FS: ظؒlatency҆ఆ • SSD storage system:
͍tail-latency 4. TAKEAWAYS FROM PRODUCTION 23 -
• QoSͷྫ • GCN (QoSߴ༏ઌ) V.S. Swift (QoS༏ઌ)ͷྫ • IOPSूதΫϥελ(IOPS-sensitive)
vs. storage RPC (throughput-intensive) • IOPSܕNIC (not fabric)ͷtail-latency͕େ͖͍ • debugʹ༗ޮ (host or fabric) 4. TAKEAWAYS FROM PRODUCTION 24 -
3. SWIFT DESIGN & IMPLEMENTATION (1/5) Loss Recovery and ACKs,
Coexistence via QoS 25 • ύέϩε • SWIFTͰtail-latency͕ྑ͍ • SACKͱRTOͰ࠷খݶͷରԠ • QoSΛͬͯSWIFTͱଞͷTCPΛڞଘΛࢦ͢ • εΠονͷQoSΩϡʔͷҰ෦ΛSWIFTʹׂΓͯ