PHPアプリケーションにおけるeBPFの使い所

PHPΞϓϦέʔγϣϯʹ͓͚Δ eBPFͷ࢖͍ॴ 2024/08/19 Sohei Iwahori (GREE, Inc.)

who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ
/ Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱ؂ࢹγεςϜ » SRE NEXT 2025ͷCo-Chairʢnew!ʣ

who?ʢ؆୯ͳྺ࢙ʣ » SIer͔Βελʔτ » PHPͰͷόοΫΤϯυ։ൃΛ݁ߏ௕͘ʢPHP5.xͷ࣌୅ʣ » ελʔτΞοϓͰΫϥ΢υΠϯϑϥ΋ؚΊͯશ෦ » ݱ৬ͰঃʑʹԼͷϨΠϠʔʹ »
ʢ8͸શવ௥͍͑ͯͳ͍͕ࠓͰ΋Կ΋ߟ͑ͣʹॻ͚Δͷ͸͓ͦΒ͘PHPʣ

ࣗࣾͷࣄۀͱPHP » ήʔϜɾΞχϝࣄۀΛத৺ͱͨ͠ϞόΠϧΠϯλʔωοταʔϏε » Ψϥέʔͷ࣌୅͔Βଘࡏ͢ΔϓϩμΫτʙݱࡏ·ͰجຊతʹPHPΛར༻ » όʔδϣϯ͸શମతʹ͸গ͠ݹΊɺ7.x͕ϝΠϯ » ྺ࢙ͷ௕͍ϓϩμΫτ΄ͲڊେͳίʔυϕʔεΛ͍࣋ͬͯΔ »
APMͱͯ͠͸NewRelic->DatadogΛར༻

Agenda » PHPerࢹ఺ͰͷPHPͷੈք » PHPͷੈքʹ͓͚ΔՄ؍ଌੑͷΧόʔൣғ » PHPΞϓϦέʔγϣϯʹ͓͚ΔeBPFͷ࢖͍ॴ » ࣮ࡍʹ࢖ͬͯΈͨ࿩ »
ࠓ౓࢖ͬͯΈ͍ͨ࿩ʢ͕࣌ؒ͋Ε͹ʣ » ·ͱΊ

PHPerࢹ఺ͰͷPHPͷੈք

PHPerࢹ఺ͰͷPHPͷੈք » ϓϩηεʢঢ়ଶʣ͸୹໋Ͱ͋Δ » ϓϩηεؒͷঢ়ଶڞ༗͸ݶఆతͰ͋Δ » PHP͸άϧʔͰ͋Δ

ϓϩηεʢঢ়ଶʣ͸୹໋Ͱ͋Δ » webͷίϯςΩετ » ϦΫΤετ୯ҐͰجຊతʹঢ়ଶ͕ഁغ͞ΕΔ » ҰఆͷϦΫΤετΛࡹ͍ͨΒϓϩηε͸࠶ੜ੒͞ΕΔ » (Α͘࢖ΘΕΔapache+mod-php΋php-fpmͳͲ) »
webҎ֎ͷίϯςΩετ » ༗໊Ͳ͜ΖͷLaravelͷΩϡʔϫʔΧʔ͸ಉ͘͡Ұఆ਺ϦΫΤετΛࡹ͍ͨΓɺϝϞϦ ࢖༻ྔ͕͖͍͠஋Λ௒͑ͨΒࣗ਎Λkillͯ͠࠶ىಈͯ͠΋Β͏Α͏ͳઃܭʹͳ͍ͬͯΔ

ϓϩηεؒͷঢ়ଶڞ༗͸ݶఆతͰ͋Δ » mod-php΍php-fpmͰ͸ͦ΋ͦ΋ϓϩηε͕෼͔Ε͍ͯΔͷͰάϩʔόϧͳঢ়ଶڞ༗͕ग़ དྷͳ͍ » apcu֦ுΛར༻ͯ͠ɺmmapͳͲͰϓϩηεؒͰڞ༗ՄೳͳΩʔόϦϡʔετΞΛར༻͢Δ ͔֎෦ͷmemcachedͳͲΛར༻͢Δ » ΞϓϦέʔγϣϯͷ಺෦ঢ়ଶΛࣗ਎ͰϝτϦΫεͰग़ྗ͢Δͷ͕ଞݴޠʹൺ΂ͯ΍΍೉͍͠ »
prometheus_client_php Ͱ͸apcu/redisͳͲΛར༻͍ͯ͠Δ1 1 https://github.com/PromPHP/prometheus_client_php

PHP͸άϧʔͰ͋Δ(1/2) https://www.php.net/manual/ja/faq.installation.php

PHP͸άϧʔͰ͋Δ(2/2) » PHPຊମ͔ΒϦϯΫ » libcryptoͳͲ » γεςϜϥΠϒϥϦʢlibxxxʣʹϦϯΫͯ͠PHPଆʹΠϯλʔϑ ΣʔεΛఏڙ͢Δ » memcached.so
+ libmemcached » curl.so + libcurl » ... » C֦ுϨΠϠʔࣗମͰͷػೳఏڙ » mysqlnd » apcu » phpredis » ...

PHPͷੈքʹ͓͚Δ Մ؍ଌੑͷΧόʔൣғ

PHPΞϓϦέʔγϣϯʢϢʔβʔίʔυʣ

APMར༻࣌(1/3)

APMར༻࣌(2/3)

APMར༻࣌(3/3)

PHPΞϓϦέʔγϣϯʹ ͓͚ΔeBPFͷ࢖͍ॴ

PHPຊମʹ࢖͍ͬͯ͘(1/2) » USDT͕࢖͑Δ2 » ΋ͪΖΜuprobe΋ » ࢖͍ॴ͸͋Γͦ͏͕ͩɺ΍΍৚͕݅໘౗ Ͱ͋ΓɺAPMͰΧόʔ͞ΕΔ෦෼Ͱ΋͋ ΔͷͰ೉͍͠ͱ͜Ζ 2
https://www.php.net/manual/ja/features.dtrace.dtrace.php

PHPຊମʹ࢖͍ͬͯ͘(2/2) » USDT͕ར༻ग़དྷΔ৚݅ » enable-dtrace͕༗ޮʹͳ͍ͬͯΔ » USE_ZEND_DTRACE=1؀ڥม਺͕ઃఆ͞Ε͍ͯΔ » ಺෦తʹ͸؀ڥม਺ΛΈͯɺτϨʔεϙΠϯτΛؚΉؔ਺ʹϙΠϯλ(execute_ex->dtrace_execute_ex) Λஔ͖׵͍͑ͯΔ3
» pidΛࢦఆ͠ͳ͚Ε͹probeΛΞλονग़དྷͳ͍ » bpftrace͸--usdt-ﬁle-activation͕͋Δ͕ɺPHPͷϓϩηεण໋͸୹͍ͷͰ௥͍͔͚ଓ͚Δ͜ͱ͸ग़དྷͳ͍ 3 https://github.com/php/php-src/blob/97b3b4552d343a830e67a63aa9b93cf4f2b1158f/Zend/zend.c#L941-L953

֦ுϨΠϠʹ࢖͍ͬͯ͘ » C֦ு΍ɺ͔ͦ͜ΒϦϯΫ͞Ε͍ͯΔڞ༗ϥΠϒϥϦΛuprobeͰτϨʔε͢ Δ » APMͰΧόʔ͞Ε͍ͯͳ͍෦෼ͰϐϯϙΠϯτʹ৘ใΛಘ͍ͨ৔߹ʹ༗༻ » ࠓճ͓࿩͢Δέʔε΋͜ͷ෦෼

֎ͷੈքͰ࢖͍ͬͯ͘ » ωοτϫʔΫ·ΘΓͷঢ়ଶ֬ೝͳͲ » strace/tcpdumpͳͲΑΓϐϯϙΠϯτͰௐ͍ࠪͨ͠έʔεͷΧόʔ » ಛʹPHPಛ༗ͷ࿩Ͱ͸ͳ͍͕૊Έ߹ΘͤͯௐࠪͰ͖Δͱศར

࣮ࡍʹ࢖ͬͯΈͨ

ෛՙࢼݧʹ͓͚ΔMemcachedͷ ߴτϥϑΟοΫݪҼௐࠪ

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ͋Δ৽نҊ݅ͰAWS্ͷ؀ڥͰෛՙࢼݧΛ࣮ࢪ͍ͯͨ͠ͱ͜Ζ͋ΔϙΠϯτ ͰϨΠςϯγ͕ѱ͘ͳͬͨ » 2୆த1୆ͷElasticache(Memcached)΁ͷτϥϑΟοΫ͕ଟ͗ͯ͢ଳҬΛ࢖͍ ੾͍ͬͯͨ » Memcached΁ͷΠϯλʔϑΣʔε͸ϑϨʔϜϫʔΫʢFuelPHPʣʹ͓೚ͤ »
෼ࢄΞϧΰϦζϜΛมߋͨ͠ΒτϥϑΟοΫͷภΔϊʔυ͕มΘͬͨ » ͦΕͧΕͷϊʔυίϚϯυͷ਺͸΄΅ಉҰͰҧ͍͸τϥϑΟοΫͷΈ

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ൃߦ͞ΕΔίϚϯυ਺͸ಉҰͳͷͰ෼ࢄࣗମ͸ग़དྷ͍ͯΔ » ঢ়گతʹ͸ಛఆͷΩʔͰڊେͳΦϒδΣΫτ͕΍ΓͱΓ͞Ε͍ͯΔΑ͏ʹݟ ͑Δ » ཪͱΓΛߦ͏ͨΊʹbpftraceͰlibmemcachedʹuprobeΛ࢓ֻ͚τϨʔεΛ औͬͯΈͨ

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ sudo BPFTRACE_STRLEN=200 bpftrace -e 'uprobe:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0:memcached_set { printf("----");time(); printf("key_length: %d\nkey:
%s\n", arg2, str(arg1)); printf("val_length: %d\nval: %s\n", arg4, str(arg3) );}'

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » fuelphpͷmemcachedυϥΠό͕memcachedʹετΞ͢ΔࡍʹΦϦδφϧͷ ΩʔΛੜ੒ͨ͠ΩʔʹϚοϐϯά » Ϛοϐϯά৘ใͷΠϯσοΫε͸୯ҰͷΦϒδΣΫτʹ֨ೲ͞Ε͍ͯͨ » ΩʔͷϚοϐϯά৘ใͳͷͰຖճࢀরɺߋ৽͞Ε͍ͯͨ » ౰֘Ҋ݅Ͱ͸Ωʔ͕૿͑΍͍͢܏޲ʹ͋ͬͨͨΊɺΦϒδΣΫτ͕ංେԽͯ͠
͍ͨ » ࠷ऴతʹϑϨʔϜϫʔΫͷػߏΛ࢖Θͣͳ͍࣮૷ʹஔ͖׵͑໰୊Λղফ

mysqlndʹ͓͚Δ caching_sha2_passwordͷڍಈௐࠪ ʢby @ts4thʣ

caching_sha2_passwordͷڍಈௐࠪ » MySQL8.0Ͱcaching_sha2_password͕σϑΥϧτʹͳΓɺ8.1Ͱmysql_native_password͕ deprecatedʹͳͬͨ » ηΩϡΞ͡Όͳ͍ܦ࿏Ͱ͸mysqld͔Βެ։伴͕औಘ͞ΕɺͦͷΩʔʹΑΓύεϫʔυ͕҉߸Խ͞Εૹ৴ ͞ΕΔ » get-server-public-keyΦϓγϣϯͰ໌ࣔతʹϦΫΤετͰ͖Δ »
σϑΥϧτͰ͸SSL_MODE=PREFERREDͳͷͰmysqlίϚϯυͳͲlibmysqlΛར༻͢Δ؀ڥ͸࠷ॳʹSSL઀ ଓΛࢼΈΔ » ରͯ͠PHPͷmysqlnd֦ு͸get-server-public-key૬౰ͷΦϓγϣϯͳͲ͕ͳ͘ڍಈʹෆ͕҆͋ͬͨ » ͦ͜ͰbpftraceͳͲΛར༻ͯ͠ڍಈௐࠪΛߦ͏͜ͱʹͨ͠

caching_sha2_passwordͷڍಈௐࠪ » mysqlnd.so΁͸ͦͷ··Ͱ͸symbolͷղܾʹࣦഊͯ͠uprobeͷΞλον͕ग़དྷͳ ͍ » σόοάγϯϘϧ php7.4-mysql-dbgsym ύοέʔδΛ௥Ճ͢Δ͜ͱͰҎԼͷؔ࿈ ͨؔ͠਺ʹΞλονͰ͖ΔΑ͏ʹͳΔ »
mysqlnd_caching_sha2_handle_server_response » mysqlnd_caching_sha2_get_auth_data » ެ։伴΁ͷΞΫηε͸libcrypto.soͷBIO_new_mem_bufؔ਺ʹΞλον͢Δ͜ͱͰ ެ։伴͕࢖ΘΕ͍ͯΔ͜ͱΛ֬ೝͨ͠

caching_sha2_passwordͷڍಈௐࠪ #ίʔυύεͷ֬ೝ sudo bpftrace -e 'uprobe:/usr/lib/php/20190902/mysqlnd.so:mysqlnd_caching_sha2_handle_server_response {printf("here\n");}' #ެ։伴Λ֬ೝ͠ɺmysqldଆ͔Βఏࣔ͞Ε͍ͯΔ΋ͷͱͷҰகΛ֬ೝ $ sudo
BPFTRACE_STRLEN=200 bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libcrypto.so.1.1:BIO_new_mem_buf {printf("%s\n", str(arg0));}' Attaching 1 probe... -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6agEXz7+WMj0uYjneXo IsWGu7pjdQB6/C3mShqTVvqnc2OgmjQ+K25qDU8MZc/yFiAVgFbQ4PFIiFIxdr2D faBBIW1fw/BMqXGRTXIa6aqDMWShcC9XNuWdsp1y3V $ sudo cat /var/lib/mysql/public_key.pem -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6agEXz7+WMj0uYjneXo IsWGu7pjdQB6/C3mShqTVvqnc2OgmjQ+K25qDU8MZc/yFiAVgFbQ4PFIiFIxdr2D faBBIW1fw/BMqXGRTXIa6aqDMWShcC9XNuWdsp1y3V1XUwWneW5R+TOmoDsx4igX ieH0xa6L8oYWM+KZNNvKD5+ESo2i9sAF+SZAgF3ZSYfET+I68KMHN/zMAR2l0Q5i YVlrKc33cAK6kabz6EeeufOzgRM6EFNrw0kMjZ5RVPsAeLO/r8NmcB4Fn7DTcpuz hkgcmbcPEAzaiI6yR0MiiDWPyBJeZgk0f96pD2iv6cpujgqWndQFc69v0OyAz7gR cwIDAQAB -----END PUBLIC KEY-----

caching_sha2_passwordͷڍಈௐࠪ » mysqlndͰ͸σϑΥϧτͷ -ssl-mode=PREFERRED ૬౰ͷڍಈͰ͸ͳ͘ɺ-- ssl-mode=DISABLED --get-server-public-key ͷΑ͏ͳಈ࡞Ͱ͋Δ͜ͱ͕ ֬ೝͰ͖ͨ »
ຊ݅͸gdbͱperfͷผղ΋͋ΔͷͰͪ͜Β΋ΑΖ͚͠Ε͹͝ཡ͍ͩ͘͞ » Debugging mysqlnd.so ʙmysql_native_password͕ഇࢭ͞ΕΔະདྷʹඋ͑ͯʙ4 4 https://labs.gree.jp/blog/2023/08/22599/

ࠓ౓࢖ͬͯΈ͍ͨ࿩ʢ͕࣌ؒ͋Ε͹ʣ

ࠓ౓࢖ͬͯΈ͍ͨ࿩1 USDTͷ׆༻ » USDTͷ׆༻ » ྫ͑͹ར༻͍ͯ͠ͳ͍ίʔυͷ͋ͿΓग़͠ͳͲ » PIDΛಈతʹ௥͍͔͚͍ͯ͘࢓૊Έ͸ඞཁɾɾ » ύεϕʔε͘Β͍Ͱ൑ఆͯ͠BPF_MAPʹೖΕͯɾɾɾ͘Β͍ͷ࡞ΓͳΒ
·͋·͋ܰྔʹग़དྷͦ͏ʁ

ࠓ౓࢖ͬͯΈ͍ͨ࿩2 ֎ͷੈ քͷ؍ଌ » TCPϨΠϠʔͷSYN࠶ૹͷՄࢹԽ » ebpf_exporterΛར༻ » libbpfʹΑΓෳ਺όʔδϣϯͷΧʔωϧ Ͱಈ࡞͍ͤͨ͞ɾɾ

·ͱΊ

» PHPݻ༗ͷΦϒβʔόϏϦςΟʹ͓͚Δ೉͠͞ͱͯ͠άϩʔόϧͳঢ়ଶอ࣋ͷ೉ ͠͞ͱ୹໋ͳϓϩηεͱ͍͏ͷ͕͋Δ » PHPͷτϨʔεϙΠϯτ͸༗༻ͦ͏͕ͩɺ্هͷ໰୊ʹΑΓ΍΍׆༻ʹ೉͕͋Δ » eBPFΛར༻͢Δ͜ͱͰC֦ுɺٴͼϦϯΫઌͷϥΠϒϥϦͷڍಈௐ͕ࠪखܰʹ ϐϯϙΠϯτͰߦ͑ͯPHP։ൃऀతʹ΋خ͍͠৔໘͸͋Δ » uprobeͰͷ֦ு·ΘΓͷௐࠪ͸Ұา໨ͱͯ͠ͱ͖ͬͭ΍͍͢ͷͰPHP։ൃऀ޲
͚ʹ΋͓͢͢Ί » ʢIDEͰͷରԠΈ͍ͨͳͷ΋͸ͳͦ͞͏ʁͳͷͰͦ͏͍ͬͨ΋ͷ͕͋ΔͱΑΓḿ Δ͔΋͠Εͳ͍Ͱ͢Ͷʣ

Thank you for listening

appendix » Tracing mysqlnd.so and mysqld (with perf and bpftrace)
» https://labs.gree.jp/blog/2023/08/22599/ » Debugging mysqlnd.so ʙmysql_native_password͕ഇࢭ͞ΕΔະདྷʹඋ͑ͯʙ » https://labs.gree.jp/blog/2023/07/22581/ » caching_sha2_password as the Preferred Authentication Plugin » https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous- series.html#upgrade-caching-sha2-password

PHPアプリケーションにおけるeBPFの使い所

PHPアプリケーションにおけるeBPFの使い所

More Decks by Sohei Iwahori

Other Decks in Technology

Featured

Transcript