Share
Embed
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
Yuuki Tsubouchi (yuuk1)
February 13, 2023
Transcript
22 slides
-
ΫϥυܕΞϓϦέʔγϣϯͷߴՄ؍ଌੑ ʹؔ͢Δݚڀ ݚڀ Studies on High Observability of Cloud
Applications ΫϥυܕΞϓϦέʔγϣϯͷ؍ଌγεςϜ ʹ͓͚Δܭଌɾอଘɾղੳͷෛՙʹؔ͢Δݚڀ A Study on Loads of Measurement, Storage, and Analysis in Observation Systems for Cloud Applications ީิᶃ ީิᶄ -
3 ใγεςϜʹ͓͚Δʮ৴པੑʯ ෳࡶͳγεςϜʹର͢Δߴසͷมߋͱߴ৴པੑΛཱ྆͢ΔͨΊͷ ֶతΞϓϩʔν͕ඞཁͱ͞Ε͍ͯΔ ݱࡏͷใγεςϜɺΠϯλʔωοτΛհͨ͠ΫϥυίϯϐϡʔςΟϯά ʹΑΔఏڙ͕ҰൠతͰ͋Δ Ϋϥυ Ϧιʔεڞ༗ɺҬωοτϫʔΫɺҟछι ϑτΣΞ/ϋʔυΣΞɺͦΕΒͷෳࡶ
ͳ૬ޓ࡞༻Λ͢γεςϜ Πϯλʔωοτ Ϋϥυܕ ΞϓϦέʔγϣϯ ઌاۀͰɺ1ෳճҎ্ͷػೳมߋ [Humble+, 2018] Accelerate: The Science of Lean Software and DevOps: Building and scaling high performing technology organizations [Beyer+, 2016] Site Reliability Engineering: How Google Runs Production Systems [Humble+, 2018] [Beyer+, 2016] -
4 2. ӡ༻ऀʹΑΓએݴ͞Εͨ·͠ ͍ঢ়ଶʹै͢Δࣗಈ੍ޚ 3. ӡ༻ऀʹΑΔखಈ੍ޚ Ϋϥυͷނোɾোʹର͢Δ੍ޚߏϞσϧ • ίϯϙʔωϯτ௨৴ϨϕϧͰͷ
ނোྼԽରԠ • ૹ੍ޚɾܦ࿏੍ޚɺࢄ߹ҙͳͲ • ܭࢉػΫϥελͷ ల։ɾࣗಈ৳ॖɾཧ • OpenStackɺKubernetes ͳͲͷΦʔέετϨʔλʔ • ৴པੑͷඪΛຬͨ͢Α͏ʹ োʹରͯ͠खಈͰରԠ • ༧ɾ༧ଌɾݕɾݪҼஅɾ ճ෮ɾࣄޙੳɾ࠶ൃࢭ ຊݚڀͰɺ3.ӡ༻ऀʹΑΔखಈ੍ޚ ʹண͢Δ Service-level Component-level System-level 1. ϓϩτίϧʹ جͮࣗ͘ಈ੍ޚ ʢϑΥʔϧττϨϥϯεʣ -
5 ੍ޚͷͨΊͷ෦ঢ়ଶͷ؍ଌٕज़ ඃ؍ଌγεςϜ σʔλετΞ ӡ༻ऀ ؍ଌγεςϜ ܭଌث ՄࢹԽ OSɾΞϓϦέʔγϣϯ
ͷܭʹΑΓܭଌՄೳ ΞϓϦέʔγϣϯ ӡ༻σʔλ ੍ޚ ઐ༻ͷ؍ଌγεςϜ͕ӡ༻σʔλΛܭଌɾอଘɾՄࢹԽ [Hauser+, CLOUD2018] ܯใ ӡ༻ऀ͕σʔλ͔Β෦ঢ়ଶΛཧղ Ͱ͖ΔΑ͏ʹ͢Δ=Մ؍ଌੑΛͨͤΔ -
6 ӡ༻σʔλࣗಈղੳٕज़ʹΑΔՄ؍ଌੑͷ্ ӡ༻σʔλετΞ ӡ༻ऀ ো༧ଌ [Notaro+, TIST2021]: A Survey of
AIOps Methods for Failure Management. [Soldani+, CSUR2022]: Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey [Notaro+, TIST2021] [Soldani+, CSUR2022] ӡ༻σʔλࣗಈղੳث োݕ োͷނোՕॴಛఆ ʢ౷ܭɾػցֶशʣ ͷ࣌ܥྻσʔλ ωοτϫʔΫ௨৴ͷґଘؔ [Kim+, PER2013] [Chen+, INFOCOM2014] [Lin, ICSOC2018] [Qiu+, Applied Science2020] [Wu+, NOMS2020] [Aggarwal+, CLOUD2021] ΞϓϦέʔγϣϯίʔυͷ վมͳ͠ͰܭՄೳͳӡ༻σʔλ ϝτϦΫε ίʔϧάϥϑ ࣌ؒɾۭؒσʔλΛ ༻͍ͨղੳख๏ -
7 ӡ༻σʔλ૿େͷ ΞϓϦέʔγϣϯ ղੳث σʔλετΞ ղੳෛՙ㽉 ܭଌෛՙ㽉 อଘෛՙ㽉 ಛʹোൃੜ࣌ʹ
࣌ؒͰͷղੳ͕ٻΊΒΕΔ ΞϓϦέʔγϣϯ͕େنԽɾෳࡶԽ͢ΔʹͭΕͯɺඞཁͳӡ༻σʔλ͕૿େ ӡ༻σʔλͷ૿େʹΑΓɺܭଌɾอଘɾղੳෛՙ͕૿େ ӡ༻σʔλͷ ࣍ݩͷ૿େ ӡ༻σʔλͷॻ͖ࠐΈճ ͱอଘྔͷ૿େ ܭଌ࣌ͷσʔλ ॲཧෛՙͷ૿େ -
8 ݚڀత ӡ༻σʔλͷ૿େͷࡍʹൃੜ͢Δɺ؍ଌγεςϜʹΑΔ ܭଌɾอଘɾղੳͷ֤ෛՙͷݦஶͳ՝Λղܾ ݅ ͘ීٴ͢Δٕज़ͷΈͷதͰ՝Λղܾ͢Δ͜ͱʹΑΓɺӡ༻ऀ ͷՃͷӡ༻ෛ୲Λܰݮ͢Δ ܭଌෛՙ อଘෛՙ ղੳෛՙ
ίʔϧάϥϑΛܭଌ͢Δ ࡍͷCPUෛՙΛݮͤ͞ Δख๏ͷఏҊ ϝτϦΫεͷૠೖෛՙͷ ݮͱอଘظؒͷظԽ Λཱ྆͢ΔΞʔΩςΫ νϟͷఏҊ ϝτϦΫεͷ૿େʹର ͯ͠ɺߴʹۭؒΛ ॖখ͢Δख๏ΛఏҊ ݚڀ՝̍ɿ ݚڀ՝̎ɿ ݚڀ՝̏ɿ -
10 [ݚڀ՝1] ίʔϧάϥϑͷܭଌෛՙ ௨৴ͷґଘΛܭଌ͢ΔͨΊʹɺ௨৴ܦ࿏Λड͢Δඞཁ͕͋Δ ιέοτϨϕϧ ύέοτϨϕϧ αʔόεΠον্Ͱ パ έοτΛड͠ɺϔομͷૹड৴ΞυϨεͱ ϙʔτ൪߸͔ΒґଘΛൃݟ͢Δ
αʔόͷOSΧʔωϧͰTCP/UDPͷ௨৴ܦ࿏ͷऴʢιέοτʣʹର͢Δ ΠϕϯτΛܭଌ͢Δ ୯ҰͷTCPଓෳͷύέοτͷϥϯυτϦοϓͰߏ͞ΕΔͨΊɺ ιέοτϨϕϧͷ΄͏͕ܭଌෛՙ͕͍ ຊݚڀͰண -
Y. Tsubouchi, et al., Low Overhead TCP/UDP Socket-based Tracing for
Discovering Network Services Dependencies, Journal of Information Processing 2022. [ݚڀ՝1] ఏҊख๏ . . . Kernel User Service Socket Tracing Process … Event Event Event ετϦʔϛϯά๏(Weave Scope) ϑϩʔू๏ ([Datadog], [SAC 20]) ϑϩʔूଋ๏ʢఏҊʣ . . . Kernel Service Socket Tracing Process . . . Event Flow Event Event Event … … . . . . . . User Service Socket Tracing Process . . . ✗ ΧʔωϧˠϢʔβۭؒؒ ͷΠϕϯτͷίϐʔίετ ✗ TCPଓϨʔτ͕૿Ճ͢Δ ͱɺίϐʔίετ͕૿Ճ ෳͷϑϩʔΛूଋ ϑϩʔ= ྆ͷΞυϨεͱϙʔτͷ ʢλϓϧʣ͕ಉҰͷ௨৴୯Ґ Event Event … … Event Event . . . Event Event … Event Event . . . Ұൠతʹ௨৴OSΧʔωϧͷTCP/UDPΛ༻͢Δ͜ͱ ʹண 11 -
12 ɾఏҊख๏ɺCPUར༻2.2%ҎԼɻ ɾϑϩʔͷ૿େʹରͯ͠ɺϑϩʔूଋʹΑΓɺ CPUར༻Λ͘ҡ͍࣋ͯ͠Δ [ݚڀ՝1] ϑϩʔͷ૿େʹର͢ΔCPU༻ྔͷมԽ ఏҊख๏ Y. Tsubouchi,
et al., Low Overhead TCP/UDP Socket-based Tracing for Discovering Network Services Dependencies, Journal of Information Processing 2022. 0 5 10 15 20 25 5 10 15 20 25 30 35 CPU usage / core (%) TCP round trips / sec (x103) Streaming(client) Streaming(server) In-Kernel-Aggr(client) In-Kernel-Aggr(server) In-Kernel-Bundling(client) In-Kernel-Bundling(server) -
13 [ݚڀ՝2] ϝτϦΫεͷอଘෛՙ ɾཧɾཧࢿݯ༻ɿCPUɾϝϞϦɾωοτϫʔΫɾσΟεΫɺϓϩηεɾεϨου ɾཁٻൃߦɿԠԆ࣌ؒɾεϧʔϓοτɾΤϥʔ ϝτϦΫεͷ σʔλετΞ ΞϓϦέʔγϣϯ ϝτϦΫεͷྫ
5-60ඵ ִؒ 300-500ఔͷ อଘظؒ ඦ͔Βेສݸͷ ܭଌର -
14 [ݚڀ՝2] ϝτϦΫεͷॻ͖ࠐΈճͱอଘྔͷߴޮԽ ϝϞϦ ϕʔεDB σΟεΫ ϕʔεDB σʔλ
ૠೖ dݸͷσʔλΛ ੵޙόονॻ͖ࠐΈ ఏҊɿೋछDBͷ֊Խ M (/s) M / d (/s) σΟεΫϕʔε ࢄDB ϝϞϦϕʔε ࢄDB ࡧҾߏ͕ฏߧ ϝτϦΫεnͱ͢Δͱ O(log n)ͷܭࢉྔ ϝϞϦ ϕʔεDB ϋογϡද O(k) ฏߧ O(log n) σΟεΫ ϕʔεDB OpenTSDB(HBase) KairosDB(Cassandra) … Redis ࡧҾߏ͕ϋογϡද O(n)ͷܭࢉྔ ॻ͖ࠐΈෛՙϝϞϦϕʔε DB͕୲͍ɺσʔλͷظอଘ σΟεΫϕʔεDB͕୲͏ ظอଘ͖ ظอଘෆ͖ -
15 [ݚڀ՝2] ܥྻͷ૿Ճʹର͢ΔૠೖεϧʔϓοτมԽ ɾσΟεΫϕʔεDBͷΈͱൺֱ͠ɺ ࠷େͰ3.96ഒͷεϧʔϓοτ ɾεϧʔϓοτͷԼվળͨ͠ 0 20 40 60
80 100 100 1K 10K 100k 1M 0 20 40 60 Insertion throughput (kilo datapoints / sec) Throughput decrease rate (%) The number of series HeteroTSDB (Proposed) KairosDB HeteroTSDB (Proposed) KairosDB ϝϞϦDB → σΟεΫDBͷҠಈεϧʔ ϓοτͱϝϞϦDBͷεϧʔϓοτͱ ಉఔ 0 20 40 60 80 100 0 300 600 900 1200 1500 1800 0 500 1000 1500 2000 Insertion throughput (kilo datapoints / sec) Memory used size (MB) Elapsed time (sec) Flushed datapoints (/sec) Memory used size (MB) -
16 [ݚڀ՝3] ϝτϦΫεͷղੳෛՙͷݮͷ՝ ɾোݕޙͷ࣌ؒͰͷނোՕॴಛఆͷͨΊʹɺଟ࣍ݩϝτϦΫεͷ࣍ݩ ݮख๏ʹண จݙ໊ ϝτϦΫε ࣍ݩݮʢख๏ʣ ނোՕॴಛఆ [PatternMatcher
21] 260K ҟৗݕʢKSݕఆʣ CNNʹΑΔ ҟৗύλʔϯྨ + ҟৗϥϯΩϯά [FluxInfer 20] 12K ҟৗݕʢࠞ߹ਖ਼نʣ ҼՌάϥϑ + PageRank [FluxRank 19] 541K ҟৗݕʢΧʔωϧີਪఆʣ ΫϥελϦϯάʢDBSCANʣ ϥϯΩϯάֶश ɾ ʮඇܧଓதͷҟৗʯʮॠؒతͳҟৗʯΛਖ਼ৗͱྨ͢Δِཅੑ͋Γ ɾ·ͨɺϝτϦΫεͷछྨ͝ͱͷௐ͕ඞཁ -
࣌ܥྻͷपظͰͳ͘ɺزԿతͳҟ ৗͷಛΛଊ͑Δख๏ͷఏҊ 17 [ݚڀ՝3] ࣍ݩݮ๏ͷϑϨʔϜϫʔΫԽ ো ݕ ނোՕॴ
ಛఆ ࣍ݩݮ ௶༎थ΄͔, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝͳஅʹ͍ͨ࣌ܥྻσʔλͷ࣍ݩݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδϜจू, 2020. ఏҊɿނোՕॴಛఆ͚ͷ࣍ݩݮ๏ΛෳͷϑΣʔζʹྨ͠ɺطଘͷख๏ ͷِཅੑͷ՝Λղܾ͢Δ ϑΣʔζ1ɿΦϑϥΠϯҟৗݕ ࣌ܥྻͷܗঢ়ͷྨࣅੑʹج͍ͮͨΫ ϥελϦϯάΛ࣮ߦ ֤Ϋϥελͷॏ৺ͱͳΔදϝτϦ ΫεΛநग़ ϑΣʔζ̎ɿܗঢ়ྨࣅੑΫϥελϦϯά -
18 [ݚڀ՝3] ࠶ݱɺϝτϦΫεݮੑೳɺߴੑͷධՁ ʮ࣍ݩݮʯˠʮނোՕॴಛఆʯͷ౷߹ධՁ ɾ࣍ݩݮͳ͠ɺطଘͷ࣍ݩݮ๏ͱൺֱ͠ɺTop-k ͷಛఆੑೳ͕ʓʓ%্ ɾϝτϦΫεΛ1k ͔Β10kʹ૿Ճͯ͠ɺطଘख๏ͱൺֱ͠ɺಛఆੑೳͷ Լ͕খ͍͞ σʔλ
ηοτ খنɿϝτϦΫε 1k ݸ େنɿϝτϦΫε 10k ݸ ʮ࣍ݩݮʯͷ෦ҐධՁ ɾϑΣʔζ̍ɿఏҊख๏ͷ࠶ݱʢRecallʣ͕97% ɾϑΣʔζ̎ɿະཧ εϥΠυ࡞࣌ͰɺطଘͷނোՕॴಛఆख ๏͕खݩͷσʔληοτͰඇৗʹ͍ಛఆੑ ೳΛࣔ͢͜ͱʹ·͞Ε͍ͯΔ -
19 ݁ ΫϥυܕΞϓϦέʔγϣϯͷՄ؍ଌੑΛ্ͤ͞Δࡍͷӡ༻σʔλྔͷ૿େ ʹର͢ΔܭଌɾอଘɾղੳͷෛՙΛݮͤ͞Δख๏ΛఏҊͨ͠ อଘ ෛՙ ղੳ ෛՙ ܭଌ ෛՙ
ɾίʔϧάϥϑͷܭଌʹண͠ɺLinuxΧʔωϧͷιέοτͷΠϕϯτΛϑϩʔ୯ ҐͰऩଋ͢ΔܭํࣜΛఏҊͨ͠ ɾಛʹTCP໋ଓ͕ଟ͍ڥͰͷCPUར༻Λݮͤͨ͞ ɾϝτϦΫεͷ૿େʹରͯ͠ɺϝϞϦϕʔεDBͱσΟεΫϕʔεDBͷࡧҾߏ ͷࠩҟʹண͠ɺೋछͷDBΛ֊Խ͢ΔΞʔΩςΫνϟΛఏҊͨ͠ ɾσΟεΫϕʔεDBͷΈͱൺֱ͠ɺ࠷େͰ3.96ഒͷεϧʔϓοτ ɾ࣍ݩʢϝτϦΫεʣͷ૿େʹରͯ͠ɺҟৗݕͱ࣌ܥྻͷྨࣅΫϥελϦ ϯάͷೋஈ֊ͷ࣍ݩݮ๏ΛఏҊͨ͠ ɾ࠶ݱΛʓʓ%ɺ˚˚%ͷ࣍ݩݮΛୡͨ͠ -
20 ത࢜՝ఔͷৼΓฦΓ ɾܭըํࣜ̍ɿ͋ΔҰͭͷఏҊͷ࣮ݱʹ͚ͯɺෳͷ࿈ଓͨ͠՝Λநग़͠ɺղܾ ɾܭըํࣜ̎ɿ࣮༻্ͷಠཱͨ͠՝Λ1ͭͣͭղܾͨ͠ޙʹڞ௨߲ͱҐஔ͚ͮΛཧ ɾຊݚڀͰɺํࣜ̎Λ݁Ռతʹ࠾༻ ɾૣظͷ࣮༻ԽΛࢦ͢΄Ͳɺղܾͷશੑ͕ٻΊΒΕΔͷͰɺ՝ಉ͕࢜ܨ͕Γʹ͍͘ ത࢜՝ఔͷݚڀܭը ɾ࡞ۀޮ ɾจݙͱ࣮ݧڥΛ՝ؒͰڞ༗Ͱ͖ͳ͍ͱɺ࡞ۀྔ͕՝͚ͩ૿େ͢Δ ɾຊݚڀͰɺશ͘ҟͳΔ࣮ݧڥΛ3ճߏஙͨ͠
ɾݚڀͷෆ࣮֬ੑͷରԠ ɾ՝͕͢ͰʹطଘݚڀͰղܾ͞ΕͨͳͲɺ్தͰ࠳ં͢Δ͜ͱ͕͋Δ ɾຊݚڀͰɺͲͷ՝ʹऔΓΉ͔Ͱ໎͏࣌ؒগͳ͔ͬͨ ܭըํࣜ̍ͷ΄͏͕͓ͦΒ͘༗ར ܭըํࣜ̎ͷ΄͏͕͓ͦΒ͘༗ར -
21 ɾ Y. Tsubouchi, M. Furukawa, R. Matsumoto, Low Overhead
TCP/UDP Socket-based Tracing for Discovering Network Services Dependencies, Journal of Information Processing (JIP), Vol.30, pp.260-268, 20223݄. ͜Ε·Ͱͷݚڀۀ: ओͳࠪಡ͖จ δϟʔφϧจ ࠃࡍձٞ ɾ Y. Tsubouchi, M. Furukawa, R. Matsumoto, Transtracer: Socket-Based Tracing of Network Dependencies among Processes in Distributed Applications, The 1st IEEE International COMPSAC Workshop on Advanced IoT Computing (AIOT 2020), July 2020. ࠃγϯϙδϜ ɾ ௶༎थ, ࡔேਓ, ᖛా݈, দխ, খྛོߒ, Ѩ෦ത, দຊ྄հ, HeteroTSDB: ҟछࢄKVSؒͷࣗಈ֊Խ ʹΑΔߴੑೳͳ࣌ܥྻσʔλϕʔε, ใॲཧֶձจࢽ, Vol.62, No.3, pp.818-828, 20213݄. ɾ Y. Tsubouchi, A. Wakisaka, K. Hamada, M. Matsuki, H. Abe, R. Matsumoto, HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores, The 43rd Annual IEEE International Computers, Software & Applications Conference (COMPSAC), pp. 264-269, July 2019. ɾ ௶༎थ, ాതจ, ݹխେ, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝͳஅʹ͍ͨ࣌ܥྻσʔλ ͷ࣍ݩݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδϜจू, 2020, 9-16 (2020-11-26), 202012݄. ɾ ௶༎थ, ҏจ, ஔాਅੜ, ࢁ૱, ദַ, ഡݪ݉Ұ, ॏෳഉআετϨʔδͷͨΊͷSHA-1 ܭࢉγεςϜͷSSE໋ྩʹΑΔߴεϧʔϓοτԽ, ిࢠใ௨৴ֶձจࢽ D, 96(10), pp.2101-2109 201310݄. ʢത࢜จʹؚΊͣʣ ɾ ௶༎थ, ੨ࢁਅ, MeltriaɿϚΠΫϩαʔϏεʹ͓͚ΔҟৗݕɾݪҼੳͷͨΊͷσʔληοτͷಈతੜ γεςϜ, Πϯλʔωοτͱӡ༻ٕज़γϯϙδϜจू, 2021, 63-70 (2021-11-18), 202111݄. -
22 ͜Ε·Ͱͷݚڀۀɿͦͷଞ ࠃߨԋ ɾ ௶༎थ, ాതจ, AI࣌ʹ͚ͨΫϥυʹ͓͚Δ৴པੑΤϯδχΞϦϯάͷະདྷߏ, ϚϧνϝσΟΞɺ ࢄɺڠௐͱϞόΠϧʢDICOMO2022ʣγϯϙδϜ, 20227݄14.
ɾ ௶༎थ, AIOpsݚڀʕSREͷͨΊͷγεςϜোͷࣗಈݪҼஅ, SRE NEXT 2022 ONLINE, 20225݄. ࠃձٞʢࠪಡͳ͠ʣ ɾ ྛ༑Ղ, দݪࠀ, ݡ, ௶༎थ, ϚΠΫϩαʔϏεܕγεςϜͷࢹʹ͓͚ΔμογϡϘʔυUIઃܭʹىҼ ͢Δঢ়گೝࣝͷӨڹ, No.2022-IOT-56, Vol.38, pp.1-8, 20223݄. ɾ দຊ྄հ, ௶༎थ, ΫϥΠΞϯτϓϩηεͷݖݶใʹجͮ͘TCPΛհͨ͠ಁաతͳݖݶํࣜͷઃܭ, ใॲཧֶձݚڀใࠂΠϯλʔωοτͱӡ༻ٕज़ʢIOTʣ, No.2020-IOT-49, Vol.11, pp.1-6, 20205݄. ɾ ྛ༑Ղ, ҏా࿇, দݪࠀ, ݡ, ௶༎थ, দຊ྄հ, ಈతదԠੑΛ࣋ͭࢄγεςϜΛରͱͨ͠γεςϜ ঢ়ଶՄࢹԽख๏ͷݕ౼, ใॲཧֶձݚڀใࠂΠϯλʔωοτͱӡ༻ٕज़ʢIOTʣ, No.2020-IOT-48, Vol.22, pp.1-8, 20203݄. ɾ ௶༎थ, ࢄΞϓϦέʔγϣϯͷ৴པੑ؍ଌٕज़ʹؔ͢Δݚڀ, SRE NEXT 2020 IN TOKYO, 20201݄25