Share
Embed
Transtracer: 分散システムにおけるTCP/UDP通信の終端点の監視によるプロセス間依存関係の自動追跡
情報処理学会 第12回インターネットと運用技術シンポジウム(IOTS)2019
論文: https://yuuk.io/papers/transtracer_iots2019.pdf
OSS: https://github.com/yuuki/transtracer
Yuuki Tsubouchi (yuuk1)
December 06, 2019
Transcript
41 slides
-
͘͞ΒΠϯλʔωοτ גࣜձࣾ (C) Copyright 1996-2019 SAKURA internet Inc ͘͞ΒΠϯλʔωοτ ݚڀॴ
Transtracer: ࢄγεςϜʹ͓͚ΔTCP/UDP௨৴ͷ ऴͷࢹʹΑΔϓϩηεؒґଘؔͷࣗಈ ใॲཧֶձ ୈ12ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδϜ(IOTS)2019 2019.12.6 ௶༎थ*1, ݹխେ*2, দຊ྄հ*1 *1) ͘͞ΒΠϯλʔωοτ, *2) ͯͳ -
2 1. എܠͱత 2. ࢄγεςϜͷґଘؔͷٕज़ 3. ఏҊख๏ 4. ࣮ݧͱߟ 5.
·ͱΊͱࠓޙͷల ࣍ -
4 WebαʔϏεͷґଘؔͷෳࡶԽͷഎܠ ظؒͷαʔϏεఏڙத ͷػೳՃ Ϣʔβʔ͔ΒͷΞΫηε ૿Ճ ୯ҰͷαʔϏεࣄۀऀ͕ ෳͷαʔϏεΛఏڙ (SNS,ECαΠτͳͲ) ༻్ಛԽͷϛυϧΣΞͷՃ
৽چγεςϜͷࠞ߹ঢ়ଶ εέʔϧΞτʹΑΔϗετ ͷ૿Ճ ෳͷαʔϏεͷҰ෦Λڞ༗ (Ϣʔβʔೝূج൫ͳͲ) ϚΠΫϩαʔϏεԽ -
6 ɾLinuxͷύέοτϑΟϧλϩάΛར༻ͯ͠L4ଓͷঢ়ଶΛऔಘ ɾLinuxΧʔωϧͷ௨৴ػߏΛར༻͢ΔݶΓɼOSϓϩηεʹରͯ͠ಁ աతʹՄೳ ɾԆΦʔόϔου: ύέοτϑΟϧλॲཧͷΦʔόϔου ɾِӄੑ: ϦιʔεফඅΛ͑ΔͨΊʹɼύέοτΛແ࡞ҝʹαϯϓϦ ϯά͢ΔͨΊɼݕ࿙ΕͷՄೳੑ͕͋Δ طଘͷࣗಈٕज़
[1] [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015. -
7 ݚڀత: ະͷϓϩηεΛෛՙͰ Linux OS Kernel Process Process TCP/UDP Flows
… . . . User ιέοτ Λࢹ TCP/UDPଓͷऴͰ͋ΔιέοτΛࢹ͠ଓΛࣗಈ 1. ԆΦʔόϔουͷղܾ ɾιέοτͷࢹΛϓϩηε ͷ௨৴ͱಠཱͤ͞Δ 2. ِӄੑͷղܾ ɾͯ͢ͷιέοτΛࢹ ɾύέοτඵؒສ୯Ґ͕ ͋Γ͑Δ ɾιέοτ࠶ར༻͞Ε͏Δ ͨΊɼιέοτͷ΄͏͕ খ͘͞ͳΓ͍͢ -
11 ࢄγεςϜͷґଘؔͷٕज़ͷྨ ϦΫΤετϕʔε Ξϓϩʔν ΞϓϦέʔγϣϯͷϦΫΤετ͕ Ͳͷ௨৴ܦ࿏ΛͨͲΔ͔Λ͢Δख๏ ίωΫγϣϯϕʔε Ξϓϩʔν ऴϗετ্Ͱτϥϯεϙʔτଓͷঢ়ଶΛ औಘ͢Δ͜ͱʹΑΓґଘؔΛ͢Δख๏
ύέοτϕʔε Ξϓϩʔν ϗετ্εΠον্ͰύέοτϔομΛ؍ଌ ͢Δ͜ͱʹΑΓɼ ґଘؔΛ͢Δख๏ -
12 ɾΞϓϦέʔγϣϯͷ֤ϦΫΤετʹࣝผࢠΛׂΓৼΓɼޙଓͷϦ ΫΤετʹຒΊࠐΜ্ͩͰɼޙଓͷϓϩηεൖͤ͞Δ ɾࣝผࢠΛཔΓʹɼϦΫΤετ͕γεςϜͷͲͷϓϩηεΛܦ༝͠ ͯॲཧ͞Ε͔ͨΛ ϦΫΤετϕʔεΞϓϩʔν M. Y. Chen, et
al., Pinpoint: Problem Determination in Large, Dynamic Internet Services, IEEE/IFIP International Conference on DSN, pp. 595–604 2002. P. Barham, et al., Magpie: Online Modelling and Performance-aware Systems, HotOS, pp. 85–90 2003. R. Fonseca, et al., X-Trace: A Pervasive Network Tracing Framework, USENIX Conference on NSDI, pp. 20–20 2007. B. H. Sigelman, et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Technical report, Google 2010. -
13 ɾLinuxͷύέοτϑΟϧλ(iptables)Λར༻ͯ͠ɼτϥϯεϙʔτ ଓΛݕग़ ɾiptablesͰଓ։࢝ϩάΛऩू͠ɼϩάίϨΫλͰதԝͷσʔλ ϕʔεʹอଘ ɾLinuxΧʔωϧͷL4௨৴ػߏΛར༻͢ΔݶΓɼґଘΛՄೳ ίωΫγϣϯϕʔεΞϓϩʔν [Clawson 15] [Clawson
15] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015. -
14 ɾωοτϫʔΫͷύέοτΛͱʹґଘΛ͢Δ ɾطଘͷτϥώοΫ͔ΒύέοτΛऩू͠ɼύέοτϔομ্ͷIPΞ υϨεͱϙʔτɼύέοτͷૹड৴ͷ࣌ࠁͳͲͷใΛղੳ͢Δ ɾ֤ܦ࿏ͰτϥώοΫྲྀྔʹ͕࣌ؒࠩ͋Δ͜ͱʹண͠ɼ֤ܦ࿏Ͱτ ϥώοΫੑ࣭ʹ૬͕ؔ͋ΔύλʔϯΛൃݟ͠ɼґଘΛਪఆ͢Δ ύέοτϕʔεΞϓϩʔν P. Bahl, et.al.:
Towards Highly Reliable Enterprise Network Services via Inference of Multi-Level Dependencies, ACM SIGCOMM Review, Vol. 37, No. 4, pp.13–24 2007. X. Chen, et.al.: Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions, USENIX Symposium on OSDI, pp.117–130 2008. P. Lucian, etl.al.: Macroscope: End-Point Approach to Networked Application Dependency Discovery, CoNEXT, pp.229–240 2009. A. Natarajan, et.al.: NSDMiner: Automated Discovery of Network Service Dependencies, IEEE INFOCOM, pp. 2507–2515 2012. A. Zand, et.al.: Rippler: Delay Injection for Service Dependency Detection, IEEE INFOCOM, pp. 2157–2165 2014. -
15 ɾϦΫΤετϕʔεɾίωΫγϣϯϕʔε ɾԆΦʔόϔου: ௨৴ܦ࿏தʹՃͷॲཧΛڬΈࠐΉΦʔόϔου ɾίωΫγϣϯϕʔεɾύέοτϕʔε ɾِӄੑ: ύέοτͷαϯϓϦϯάʹΑΓྲྀྔͷখ͍͞ଓΛݕͰ ͖ͳ͍ ɾ୯Ґ: OSϓϩηεͰͳ͘IPΞυϨε୯ҐͱͳΔ
ɾϦΫΤετϕʔε ɾܭଌ४උίετ: طଘͷΞϓϦέʔγϣϯʹॲཧΛՃ͢Δ खؒ ֤Ξϓϩʔνͷ՝ -
17 ఏҊख๏: ιέοτϕʔεΞϓϩʔν Linux OS Kernel Process Process TCP/UDP Flows
… . . . User ιέοτ Λࢹ ɾιέοτͷࢹϓϩηεͷ ௨৴ͱಠཱ͍ͯ͠ΔͨΊɼ ԆΦʔόϔουΛݮՄೳ ɾଓ࠶ར༻࣌ʹͯ͢ͷι έοτΛෛՙͰࢹՄೳ ɾιέοτʹΑΓɼϓϩηεͱ ଓͷඥ͚͕Մೳ ɾϓϩηε͕ιέοτΛར༻͢ ΔݶΓಁաతʹՄೳ -
18 TranstracerͷγεςϜߏ Host 1 Host 2 Host N CMDB Tracer
Tracer Tracer Systems Administrator ɾϗετ্ʹTracerϓϩηεΛஔ ɾ֤Tracerϓϩηεऔಘͨ͠ଓ ใΛCMDB(ଓใཧDB)ʹอଘ ɾγεςϜཧऀCMDBʹΞΫηε ͠ɼෳͷϗετʹ·͕ͨΓґଘؔ Λऔಘ -
19 ଓใͷऔಘ Host Kernel Process Process TCP/UDP Flows … Tracer
Polling ɾTracerϓϩηε͕LinuxΧʔωϧʹ͍߹Θ ͤɼTCP/UDPιέοτใΛϙʔϦϯάऔಘ ɾଓΛऴ͢ΔOSϓϩηεใ͋Θͤͯ औಘ ɾιέοτใ: /proc/net/tcpNetlink sock_diag ɾϓϩηεใ: /proc/<pid>/{stat,fd} . . . ॲཧʹհೖ͠ͳ͍ͨΊ Φʔόʔϔου -
20 Transtracerͷར༻ྫ $ ttctl --dbhost 10.0.0.20 --ipv4 10.0.0.10 10.0.0.10:80 (’nginx’,
pgid=4656) ᵋ<-- 10.0.0.11:many (’wrk’, pgid=5982) 10.0.0.10:80 (’nginx’, pgid=4656) ᵋ--> 10.0.0.12:8080 (’python’, pgid=6111) 10.0.0.10:many (’fluentd’, pgid=2127) ᵋ--> 10.0.0.13:24224 (’fluentd’, pgid=2001) 10.0.0.10 nginx 10.0.0.11 wrk 10.0.0.12 python 10.0.0.13 fluentd :80 fluentd :8080 :24224 -
21 ϓϩηεͷσʔλߏ ᶃ LinuxͷϓϩηεάϧʔϓΛ ϊʔυͷ࠷খ୯Ґͱ͢Δ ᵓᴷnginx,627,627 ᴹ ᵓᴷnginx,628,627 ᴹ ᵋᴷnginx,629,627
$ pstree -apg | grep nginx ᶄ (ipv4, pgid, pname)Ͱϓϩηε ʹҰҙ੍Λ͔͚Δ ϓϩηε࠶ىಈ͢ΔͱID͕มԽ ͯ͠͠·͏ͨΊɼ͍߹Θͤ࣌ʹ pgid͕ҟͳΔͷΛॏෳഉআ -
22 ଓཧͷͨΊͷσʔλߏ ᶅ ActiveͱPassiveʹϊʔυΛྨ ᶆ Active => PassiveͷϑϩʔΛอଘ Active Passive
Process Passive Active Port N Port M Active ಉҰϓϩηε͕Activeʹ PassiveʹͳΓ͑Δ ಉҰϓϩηε͕ෳͷϙʔτ ΛϦοεϯ͢Δ͜ͱ͕͋Δ -
25 1. ΞϓϦέʔγϣϯʹ༩͑ΔԠԆΦʔόϔουͷධՁ 2. ଓใऔಘͷͨΊͷCPUར༻ΦʔόϔουͷධՁ ධՁ࣮ݧ ఏҊख๏ͱಛ͕ࣅ͍ͯΔίωΫγϣϯϕʔεΞϓϩʔνͷઌߦख๏[1] ͱ֤ΦʔόϔουΛൺֱ [1] J.
K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015. -
27 ࣮ݧڥͷৄࡉ ߲ ༷ Client CPU Memory Benchmarker Intel Xeon
CPU E5-2650 v3 2.30GHz 2core 1 GB wrk 4.1.0-4 Server CPU Memory HTTP Server Intel Xeon CPU E5-2650 v3 2.30GHz 4core 1GB nginx 1.17.3 CMDB CPU Memory Database Intel Xeon CPU E5-2650 v3 2.30GHz 1core 1 GB PostgreSQL 11.3 ɾΠϯελϯεͯ͘͢͞ΒͷΫϥυ্ʹߏங ɾLinux Kernel 4.15 (Ubuntu Server 18.04.3 LTS) -
28 1. Normal: ॲཧ͕ͳ͍ঢ়ଶ 2. Transtracer: ఏҊख๏ (https://github.com/yuuki/transtracer v0.1.0) ɾϙʔϦϯάִؒ1ඵ
3. iptables NEWϑΟϧλํࣜ: ৽نଓͷΈϩάΛग़ྗ 4. iptables ESTBϑΟϧλํࣜ: ଓཱ֬தʹΓͱΓ͞ΕΔύέοτ ͷϩάΛαϯϓϦϯάͤͣʹग़ྗ ɾઌߦख๏ͷແ࡞ҝͳαϯϓϦϯάͰɼ໋ͳଓͰ͋ͬͯྲྀ ྔ͕খ͍͞ଓΛݟಀ͢Մೳੑ͕͋Δ ࣮ݧʹ༻͍Δ֤࣮ -
29 ԠԆΦʔόʔϔου 50 100 150 200 250 300 350 400
450 500 5000 10000 15000 20000 Average Latency (ms) Connections Normal 93.1 191.6 279.3 353.8 Transtracer 94.7 188.3 291.8 401.2 ESTB filter 115.0 236.0 359.0 462.5 NEW filter 113.1 214.4 310.0 449.3 ɾNormalʹରͯ͠transtracer ͕1.7~13.4%ͷΦʔόϔο υ૿ ɾiptables࣮ͷESTBϑΟϧ λํࣜʹରͯ͠ɼ transtracer͕13-20%ͷ Φʔόϔουݮগ -
30 CPUར༻Φʔόʔϔου 0 10 20 30 40 50 60 70
80 90 100 5000 10000 15000 20000 0 50 100 150 200 250 300 350 400 450 500 CPU usage (%) Reading sockets time(ms) Connections ttracerd’s CPU usage 13.2 23.0 34.2 44.4 ESTB filter’s CPU usage 72.2 75.9 78.8 78.6 Reading sockets time 102.3 199.1 317.8 408.6 ɾ20,000ଓʹ͓͍ͯɼ TranstracerͷCPUར༻ 44.4%ɼESTBϑΟϧλํ ࣜͷCPUར༻78.6% ɾ43.5%ͷCPUར༻ͷ ݮ -
31 ɾ2ສଓ࣌ʹϙʔϦϯάؒ ִΛ૿Ճͤ͞ΔͱCPUར༻ ͕Ͳͷఔݮ͞ΕΔ͔ ɾ5ඵҎͷ໋ͳଓΛݕ ग़Ͱ͖ͳ͘ͳΔՄೳੑ͕͋ Δ͔ΘΓʹɼ8.6%·Ͱݮ Մೳ ϙʔϦϯάִؒͱCPUར༻ͷؔ 0
5 10 15 20 25 30 35 40 45 50 55 1 2 3 4 5 CPU usage (%) Polling interval CPU usage 44.4 21.6 13.0 10.8 8.6 -
39 ɾϙʔϦϯά͕ݟಀ໋͢ͳ ଓΛݕग़͢ΔͨΊʹετ ϦʔϛϯάΛΈ߹ΘͤΔ ɾeBPFʹΑΓɼconnect(2)ͱ accept(2)ΠϕϯτΛऔಘ ͠ɼϑϩʔใΛऔಘ͢Δ ɾUDPͷ߹send_msg(2), recv_msg(2)Πϕϯτ ετϦʔϛϯάʹΑΔଓͷݕग़
Linux Host Kernel Process Process TCP/UDP Flows … . . . User Streaming Tracer -
40 ɾiovisor/bccͷtcpacceptͰɼඇӬଓԽڥͰͷෛՙ࣮ݧ ɾwrk (HTTP KeepAlive off)Ͱಉ࣌ଓ1000ͰnginxʹϕϯνϚʔΫ ɾCPUར༻45~50%/coreఔ ɾԠԆͷΦʔόϔου༗ҙͳѱԽݟΒΕͳ͔ͬͨ ɾeBPFͷΠϕϯτΛͯ͢Ϣʔβʔϥϯυʹίϐʔ͍ͯ͠Δ͜ͱ͕ CPUෛՙ͕ߴ͍ཁҼ͔͠Εͳ͍
eBPFͷετϦʔϛϯάෛՙͷ༧උ࣮ݧ ৄࡉ࣮ݧϊʔτ https://www.notion.so/yuuk1/iovisor-bcc-tcpconnect-tcpaccept-af2d1fdce35c49fb945b548db373213d -
41 ɾͯ͢ͷଓใΛऩू͢ΔͱɼCMDBʹ֨ೲ͢Δσʔλྔ͕େ͖͘ ͳΔͨΊɼͳใΛݮ͢Δ ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓͯΒΕΔϥϯμϜͳૹ৴ݩ ϙʔτ ɾಛఆͷLISTENϙʔτෳͷΤϑΣϝϥϧϙʔτ͔Βଓ͞ΕΔ ɾ͜ΕΒͷଓΛू͠ɼ୯ҰͷଓͱΈͳ͢ ΤϑΣϝϥϧϙʔτͷू Host
Port Process Port Port Host Port Process ୯Ұͷଓ ͱͯ͠ू LISTEN ΤϑΣϝϥϧ