Share
Embed
#51 “Empowering Azure Storage with RDMA”
NSDI '23
- https://www.usenix.org/conference/nsdi23/presentation/bai
- https://www.microsoft.com/en-us/research/publication/empowering-azure-storage-with-rdma/
cafenero_777
March 24, 2024
Transcript
23 slides
-
Research Paper Introduction #51 “Empowering Azure Storage with RDMA” ௨ࢉ#128
@cafenero_777 2024/03/21 1 -
ରจ •Empowering Azure Storage with RDMA • Wei Bai, et
al • Microsoft • NSDI '23 • https://www.usenix.org/conference/nsdi23/presentation/bai • https://www.microsoft.com/en-us/research/publication/empowering-azure- storage-with-rdma/ 2 -
Agenda •ରจ •֓ཁͱಡ͏ͱͨ͠ཧ༝ 1. Introduction 2. Background 3. Overview 4.
Storage Protocols over RDMA 5. RDMA Estats 6. Switch Management 7. Congestion Control 8. Experience 9. Lessons and Open Problems 10.Related Work 11.Conclusions and Future Work 4 -
֓ཁͱಡ͏ͱͨ͠ཧ༝ •֓ཁ • Azure ͰVM/Storage, Storage/StorageؒͰRDMAΛར༻ɻRegionͰଓɻ • RegionRDMAͷෳࡶੑinter-operabilityͳͲͷܦݧͷڞ༗ • τϥϑΟΫͷ70%ΛΊΔRDMA௨৴ͷDisk
I/Oੑೳ্ͱCPUઅΛ࣮ݱ •ಡ͏ͱͨ͠ཧ༝ • ҎલNSDI '23ΛࣼΊಡΈͯ͠ؾʹͳͬͨͷͰɻ • RDMA in Production • SONiCΛ͍ͬͯΔΒ͍͠ 5 -
RDMA? RoCEv2? •Remote DMA(Direct Memory Access) • σʔλసૹԆͱCPUෛՙͷݮ •RoCEv2: RDMA
over Converged Ethernet v2 • Ethernet NWΛ༻͍ͯɺRDMA௨৴Λ࣮ߦ • EthͳطଘڥΛ࠶ར༻͍͢͠ɻୠ͠loss͕ൃੜ͢Δͱੑೳܹམͪɾɾɾ • Ԇ͕ٻΊΒΕΔͷʢετϨʔδؒGPUؒ௨৴ͳͲʣʹ׆༻ 6 -
1. Introduction •SSDશ࣌Ͱಉ༷ͷੑೳ͕ٻΊΒΕΔ •ετϨʔδ on NW • Clos NWͰଳҬ͋Δɻ͕ɺCPUॲཧʹΑΔԆCPUෛՙ͕՝ •
͡Ό͋RDMAͰΦϑϩʔυͤ͞Α͏ •VM/Storage cluster, Storage clusterؒͰRDMA • Intra-region: ಉ͡Regionͷҧ͏DCʢҧ͏ݐʣ •Complexity and heterogeneity • DCQCNαϙʔτ͍ͯ͠Δ͕࣮͕ҟͳΔ • ҟͳΔεΠονϕϯμʔͷ࣮ࠩҟ • DCؒͷڑଓͰRTT૿Ճ •ΞϓϦʢL7ʣ͔ΒL2·Ͱमਖ਼ͱ࠷దԽΛ࣮ࢪɻϝτϦΫεػೳͷ֦ॆ͠statusࢹɻ •SONiCར༻ͰιϑτΣΞελοΫΛ౷ҰɻDCQCN/PFCͰߴεϧʔϓοτɾϨΠςϯγɾ΄΅θϩϩεΛୡ •2018: όοΫΤϯυΛRDMAԽ, 2019: ϑϩϯτΤϯυΛRDMAԽ 7 -
2. Background (1/2) •Clos NW/ECMP •VM/StorageΫϥελ •PS, ENͷϨΠϠʔߏɻEN3ʢӈਤʣ • Write:
ΞΫςΟϒͳextentʹappend͠ɺͯ͢ͷ EN͔ΒԠΛड৴ͯྃ͠ • Read: ҙͳextentΛಡΈࠐΈɺϨεϙϯε͢Δɻ • ͦͷଞ: Garbage collection, erasure coding 8 •T0 (ToR) •T1: T0 clustera •DC: T2Ϋϥελ •Region: RHʹଓ͞ΕͨLong-haul (~10km) N [us] 2 [ms] boxܕSW Shallow bu ff er chassisܕSW Deep bu ff er/VoQ { { { HV Disk driver͕ I/OΛrequest incast ى͖͕ͪ -
2. Background (2/2) •RDMA࠾༻ϝϦοτ • NVMe/SSDಉੑೳͷཁٻɻඦus latency, ेGbpsଳҬ • TCP/IPελοΫར༻ʹΑΔॲཧԆͱCPUෛՙ
• RDMAར༻: HWར༻ʹґΔԆ(us)ͱߴεϧʔϓοτɺCPUΦϑϩʔυ • VM/StorageؒRDMAϦʔδϣϯͰͷߦ͏ඞཁ͋Γ •՝ͱઓ • طଘࢿ࢈ͷྲྀ༻: RoCEv2 (RDMA over commodity Ethernet v2)ར༻ͱPFC (Priority-based Flow Control)ར༻ʹΑΔύέϩεճආ • ϦʔδϣϯRDMAʢΫϥελؒRDMAʣͰͷઓ • ҟͳΔNICͷར༻ʹΑΔDCQCN࣮ͷҧ͍ɻGen1-3 • ҟͳΔεΠονHW/OS: όοϑΝΞʔΩςΫνϟαΠζɾϝΧχζϜɾࢹํ๏͕ҟͳΔɻӡ༻େมɻ • T2-RHؒͰͷԆ: us~2ms·ͰΒͭ͘ɻPFCͰ·͔ͳ͑Δʁ • ϝτϦΫεͱࣗಈϑΣΠϧΦʔόʔ 9 ͍&༧ظͰ͖ͳ͍ʂ -
3. Overview •sU-RDAM, sK-RDMA • RDMAϕʔεͷϓϩτίϧɻFEʗBE௨৴Λαϙʔτ •RDMA Estats: RDMAૢ࡞ͷਖ਼֬ͳstatsఏڙ •NWPFC/DCQCNΛར༻͠ɺߴεϧʔϓοτɾϨΠςϯγɾύέϩε΄΅θϩ
•ιϑτΣΞελοΫΛ౷Ұ͢ΔͨΊSONiCΛಋೖ •NICͷϑΝʔϜΣΞߋ৽͠ɺDCQCNͷಈ࡞Λ౷Ұ •PFCετʔϜͷ؇ʢPFC watchdogʣ 10 -
4. Storage Protocols over RDMA (1/2) •sU-RDMA • ετϨʔδؒ௨৴ͷRPCϓϩτίϧελοΫ •
Socket-like API on User-space • Memory (re-)allocation • ड৴ଆͷཁٻड͚ೖΕpre-post • ૹड৴ଆํͷσʔλαΠζͷ߹ҙ • CreditʹԠͯ͡ૹ৴ํ๏Λมߋ͠ɺόοϑΝͷ༗ޮ׆༻ɻ • খ: RDMAૹड৴Λͦͷ··ར༻ • த: WriteͱWrite/Write DoneΛૹ৴ • େ: SendͰૹΔɻRead, Read DoneΛฦ͢ɻ • Legacy/fail-over༻ʹTCPར༻Մೳ 11 socket-like API -
4. Storage Protocols over RDMA (2/2) •sK-RDMA • VM/ετϨʔδؒ௨৴ͷRPCϓϩτίϧελοΫ •
Socket-like API on kernel-space • ಉ༷ʹcredit-based, PFC, RDMA/TCP • σʔλసૹετϨʔδଆ͔Β࢝·Δʹҙ 12 ੨ࣈίϯτϩʔϧϝοηʔδ ࣈσʔλϝοηʔδ -
5. RDMA Estats •Estats: RDMA Extended Statistics • ΤϯυϗετଆͷৄࡉͳϝτϦΫεՄࢹԽ •
TCP netstatͷΑ͏ͳͷ • Tx, RX byte, NACKͳͲɻ • ֤ϨΠςϯγΛώετάϥϜԽ͠ޭɾࣦഊΛੳ • ߴϨΠςϯγൃੜ࣌ʹNIC/QP dumpػೳͱղੳ 13 PCIe NW T1: ૹ৴Ωϡʔͷൃߦ T5: NICͰड৴࣌ T6: CPUͰड৴࣌ T6-T1: total operation latency T5-T1: NIC/NW latency -
6. Switch Management •ҟͳΔϕϯμʔͷҟͳΔεΠον: • ༷ʑͳཁٻͷͨΊٕज़ਐԽ͕͍ɻҟͳΔASICͷҟͳΔbu ff er arch. •
SONiC: SAIͰͷநԽɻΫϩεϓϥοτϑΥʔϜͱͯ͠ར༻͠ૢ࡞ɾཧͷ౷Ұ •SONiCͱRDMA • ඞཁͳ֤ػೳ͕͋Δ: ECN marking, PFC, PFC watchdog shared bu ff er model • Shared bu ff erΛ༻్Ͱׂར༻ɻόοϑΝαΠζ͖͍͠Λࢦఆ: lossless༻PFCൃߦͳͲ • Shared 18MB, 6MB for lossless headroom •Testing • PTF: Packet Testing FrameworkΛ༻͍ͨػೳςετ • breakpoint/dumpΛ༻͍ͨૹड৴Ұக֬ೝͰbu ff er/counter/schedΛςετ • HWͱ༻͍ͨus/nsͳtra ff i c generatorͰRED/ECN markingPFC watchdogػೳςετɻ͜ͷลΓʁ 14 boxܕSW Shallow bu ff er { -
7. Congestion Control (1/2) •PCF/DCQCNΛ༻͍ͯLong linkͳregionalͰscaleͰ͖ΔΑ͏ʹ͢Δ • ͯ͢ͷϙʔτ͕ಉ࣌ʹ͢ΔέʔεΛߟ͑ͳ͍ • T2/RHo
ff chip DRAMΛdeep bu ff erͱͯ͠ར༻ɺRDMAΛड͚Δ • PFC headroomQຖͰͳ͘switch ingressຖʹ੩తᮢ(Φʔόʔαϒ)Ͱٵऩɻ • ܦݧଇʹΑΔɻϩεഉআͱόʔετੑ্͕ •DCQCN interoperability • QP͝ͱͷૹ৴੍ޚʹDCQCNΛར༻ɻ • ૹ৴ऀ/RPɺεΠον/CPɺड৴ऀ/NP • NIC࣮ͷҧ͍: • Gen1: CP/NP statemachineFirmware࣮ͷͨΊɺCNPൃੜΛ࠷খԽɻ • cache missൃੜ͠қ͍ͨΊɺper packetͰͳ͘packet burst୯ҐͰrate limit • Gen2/3: DCQCN HW࣮ɻECNຖʹCNPൃߦ͢Δɻ୯Ґ࣌ؒຖͷCNPͰper packet rate limit • Gen1/23ΛࠞͥΔͱcache miss࿈ൃͰੑೳྼԽ + Gen1ʹաͳCNPΛૹΓɺաʹrate limit͕ൃੜ • Gen2/3ଆͰQPຖʹCNP limitterͰ࿈ଓCNPൃߦΛgen1ͷִؒʹ߹Θͤͨɻ·ͨRPଆ୯Ґ࣌ؒGen1ಉʹௐɻ • Gen2/3Λburstຖʹrate limit͢ΔΑ͏มߋ 15 N [us] 100~m 2 [ms] 10~km boxܕSW Shallow bu ff er chassisܕSW Deep bu ff er/VoQ { { ૹ৴ऀ RP εΠον CP ड৴ऀ NP ECN CNP Reduced rate! Packet ECNΛ༻͍ͨύέοτϑϩʔྫ -
7. Congestion Control (2/2) •DCQCNͷνϡʔχϯά • DCQCNͷάϩʔόϧύϥϝʔλͷΈมߋՄೳ • RDMAͷϑϩʔ2छྨͷΈΛαϙʔτ͍ͨ͠ʢfrontend, backendʣ
• ϞσϧཧͰಛੑཧղ -> lab testͰଥͳύϥϝλਪఆ -> ຊ൪ಉ༷ ͷςετڥͰύϥϝʔλܾఆ • DCQCNrate-baseͷͨΊRTTͷunfairnessແ͍ • RTT͕େ͖͍ͷʹߴ͍εϧʔϓοτΛͤΔʹେ͖ͳKmax- Kmin or খ͞ͳPmaxʹsparse ECN marking͢Δ • DCQCNͱεΠονόοϑΝҰॹʹνϡʔχϯά͢Δඞཁ͋Γɻ KminΛେ͖͘͢ΔલʹόοϑΝΛᮢΛେ͖͘͢ΔͳͲɻ 16 σʔληϯλɾ֎Ͱ͚Δɺͱ͠ͳ͔ͬͨɻ : 2us : 1.77ms Lab testͷ݁Ռ -
8. Experience Timeline and deployment •Timeline • 2018: Backend (Inter-Storage)
• 2019: Frontend (Storage/Compute) • 2020: Intra-region RDMA • 2023/02: ALL region supported •ຊ൪ڥͷಋೖͷεςοϓ • Component testing, Same stack testing in testbed • ຊ൪ڥʹগͣͭ͠RDMAΛಋೖ w/ NIC driver/ fi rmware update, NOS update • T0εΠονSONiCͷfast reboot, warm rebootΛར༻͠1ඵະຬͷ௨৴அ • NICυϥΠόߋ৽TCPϞʔυʹҠߦޙʹ࣮ߦ 17 -
8. Experience 2018ͷA/Bςετ݁Ռ •backend • CPU༻ʢstorage App, Storage protocol, TCP/IP
stackʣ • Backend storage tra ffi c FCT 18 TPSͷߴ͍workloadΛ࣮ߦ •Frontend •SiskSpdͰI/OαΠζ8KBͰIOPS 2ύλʔϯͰςετɻCPU ࠷େ34.5%ݮ •࣮ڥVMΛ༻͍ͨςετɻTCP, RDMAͷฏۉϨΠςϯγͷൺֱɻେ͖ͳI/Oͩ ͱ20%Ҏ্ݮCPU༻ʢstorage App, Storage protocol, TCP/IP stackʣ ff -
8. Experience ݟ͔ͭͬͨ՝ͱमਖ਼ •FMR Hidden fence • RegionରԠ͕ͨ͠ۃʹεϧʔϓοτ͕͍ɾɾɾ • Compute
nodeͰͷsK-RDMAར༻࣌ʹSendΛૹΔ • ͕ɺಛఆRTT (In fl ight)Ͱ1ͭͷSend͔͠ૹΕͳ͍༷ • ϕϯμʔଆͰυϥΠόमਖ਼ •PFCͱMACsec • ڑ௨৴ͷΈԿނ͔drop͕ଟ͍ -> PFC frame͕drop͞Ε͍ͯΔɾɾɾ • PFCͷ҉߸Խʹؔ͢Δ༷͕ଘࡏ͠ͳ͍ɾɾɾͷͰɺඪ४Խͨ͠ 19 -
9. Lessons and Open Problems •Failoverίετߴ͗͢ • ࠷ޙͷखஈͱͯ͠sU/K-RDMAʹ࣮ɻ͏=ॏେΠϯγσϯτ • CPU
core util/loadͷര૿ͱރׇɻͭ·ΓϦεΫɻσϓϩΠ৻ॏʹʂ •ϗετؒNWͱཧεΠονͷ౷߹ • ϗετʢE-WʣͱϗετؒʢN-SʣɻN-SϦιʔεফඅେɻকདྷͲ͏͢Δʁ •εΠονόοϑΝ • ༰ྔͩͱੑೳΛҾ͘ɻDCQCNͰͳ͘εΠονύϥϝʔλͰղܾɻ • ͳͥʁ -> όʔεττϥϑΟοΫରࡦɻDCͰ͋Γ͕ͪɻ • Deep bu ff erඞཁɻΑΓϞμϯͳbu ff erཧϝΧχζϜʢϓϩάϥϚϒϧʁʣ͕΄͍͠ɻ 20 -
10. Related Work •Deployment experience of RDMA • Bing •
Alibaba (LUNA/SOLAR: Storage backend. user-space TCP, UDP on DPU) • AWS SRD (Scalabe Reliable Datagra) on Nitro NIC for HPC/ML/Storage •RDMAվળϙΠϯτ in DC • CC: ECN, delay-based, int-based, credit-based • ্هҎ֎: dead-lockճආɺmulti-pathαϙʔτɺηΩϡϦςΟɺԾԽɺmulti-tenant •ߴԽͷಓ۩ • Socket, Kernel, SmartNIC 21 -
11. Conclusions and Future Work •intra-region RDMA in Azure •
70% RDMA • ͯ͢ͷpublic regionͰRDMAαϙʔτ • Disk I/O performanceվળͱCPUίΞઅΛಉ࣌ʹ࣮ݱ •γεςϜΞʔΩςΫνϟɺHW acceleration, CCͰվળ༧ఆ 22