Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BPF/bccによるトレーシング入門 (仮)
Search
mmisono
November 05, 2018
Programming
5
2.1k
BPF/bccによる トレーシング入門 (仮)
2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/
mmisono
November 05, 2018
Tweet
Share
More Decks by mmisono
See All by mmisono
Tracing BitVisor with bpftrace
mmisono
0
680
vIOMMU implementation in BitVisor
mmisono
0
520
bitvisor.ko : BitVisor as a module
mmisono
0
740
BPFを利用したBitVisor内部でのパケットフィルタリング (+α)
mmisono
2
960
Other Decks in Programming
See All in Programming
Amazon Neptuneで始めてみるグラフDB-OpenSearchによるグラフの全文検索-
satoshi256kbyte
4
330
Identifying User Idenity
moro
6
7.8k
Tuning GraphQL on Rails
pyama86
2
1k
Universal Linksの実装方法と陥りがちな罠
kaitokudou
1
220
CSC509 Lecture 08
javiergs
PRO
0
110
Vue3の一歩踏み込んだパフォーマンスチューニング2024
hal_spidernight
3
3.1k
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
830
Vitest Browser Mode への期待 / Vitest Browser Mode
odanado
PRO
2
1.7k
リリース8年目のサービスの1800個のERBファイルをViewComponentに移行した方法とその結果
katty0324
5
3.6k
NSOutlineView何もわからん:( 前編 / I Don't Understand About NSOutlineView :( Pt. 1
usagimaru
0
110
Why Spring Matters to Jakarta EE - and Vice Versa
ivargrimstad
0
970
Go言語でターミナルフレンドリーなAIコマンド、afaを作った/fukuokago20_afa
monochromegane
2
140
Featured
See All Featured
A Philosophy of Restraint
colly
203
16k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
Become a Pro
speakerdeck
PRO
24
5k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
The Cost Of JavaScript in 2023
addyosmani
45
6.6k
Music & Morning Musume
bryan
46
6.1k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Rails Girls Zürich Keynote
gr2m
93
13k
Teambox: Starting and Learning
jrom
132
8.7k
Adopting Sorbet at Scale
ufuk
73
9k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Transcript
#1'CDDʹΑΔ τϨʔγϯάೖ Ծ 2018/11/5 OSS
͜ͷൃදͷ༰ 2 BPFʹΑΔτϨʔγϯάͷ෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ͍ํͷઆ໌ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
1 Linux Tracing System 4
͡Ίʹ 5 BPFͰͷτϨʔγϯά = º શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏ͷ
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic
Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace
Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz
σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝ͰใΛऔಘ ɾ*1$ Ωϟογϡώοτ ʜ Performance Counter (PMU)
ɾ4UBUJD5SBDJOH ɾΧʔωϧʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔΛొͰ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔΛొͰ͖Δ Kprobe 1 2 3
1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ *1$ Ωϟογϡώοτ ذ༧ଌώοτ ʜ ɾ*OUFMͷ߹
ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ ɾͲͷใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷʹୡͨ͠߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸҎ্ͷใΛऔಘ͍ͨ͠߹͏·ׂ࣌͘͢Δඞཁ͕͋Δ
5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε ͍͍ͩͨ5SBDFQPJOUͷఆٛ
ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯΠϯ λϑΣʔεతͳޓੑ͕͋Δʢͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
,QSPCF 11 Insn Break point pre handler post handler Insn
( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧͷେ෦͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
ɾ-JOVYඪ४උͷϓϩϑΝΠϥ ɾΧʔωϧϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ
ɾΠϕϯτͷൃੜճͷΧϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧΛར༻ 1FSG DGQFSGGUSBDFͷΈ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
2 Tracing with BPF
5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event
Call BPF Program Helper Function pid uid … eBPF Map perf buffer
15 bpf(2) system call Create BPF map Kernel Userland BPF
map User Program
16 bpf(2) system call Verifier C source BPF Program JIT
(Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
17 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
18 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
#1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ݁ՌΛऔಘ ɾ݁ՌʹΛͯ͠NBQʹॻ͖͢ Πϕϯτൃੜճͷܭ ɾϖΞͱͳΔؔΛݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔͰ࣌ࠁΛऔಘɼNBQʹ֨ೲ
ɾFQJMPHVFͷؔͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
#1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (
( ( * , * ,( * ( ( ( *) ( * (
Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF
6QSPCF 5SBDFQPJOU 1FSGTPGUXBSF IBSEXBSFFWFOU https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
#1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
3 Tracing with bcc
#1'ϓϩάϥϜ࡞Ͱେมͳ • υΩϡϝϯτෆɼγεςϜίʔϧͷཧղ͕େม • CQG QFSG@FWFOU@PQFO ͱ͍͏ڧఢ •
$ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆελοΫʹஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞ͨ͠#1'ϓϩάϥϜ&-'όΠφϦ • &-'όΠφϦΛదʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͍ʹ͍͘
CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ τϨʔγϯάʹݶఆ͢ΔͷͰͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
1ZUIPO (P ˞τϨʔγϯάίʔυࣗମ$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
CDDͰͷϓϩάϥϜྫ 26 finish_task_switch() kprobe Python map
! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
.PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'
7+19 &… &Clang$%%)5 :%#* AST & eBPF map/8 &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
CDDDPNQJMFS 28 ) : : (
πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&
ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
πʔϧͱͯ͠ͷCDD 30
πʔϧͱͯ͠ͷCDD 31
CDDͷܽ ༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹجຊతʹҾͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ໘ଟʑ͋Δ ґଘ͕ؔ૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
Complementary 33
ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply
(Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
(P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf
loader (elfόΠφϦࣗͰίϯύΠϧ͢Δ)
ຊઆ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿΒͳͯ͘)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38