Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BPF/bccによるトレーシング入門 (仮)
Search
mmisono
November 05, 2018
Programming
5
2.1k
BPF/bccによる トレーシング入門 (仮)
2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/
mmisono
November 05, 2018
Tweet
Share
More Decks by mmisono
See All by mmisono
Tracing BitVisor with bpftrace
mmisono
0
720
vIOMMU implementation in BitVisor
mmisono
0
550
bitvisor.ko : BitVisor as a module
mmisono
0
810
BPFを利用したBitVisor内部でのパケットフィルタリング (+α)
mmisono
2
1k
Other Decks in Programming
See All in Programming
私の愛したLaravel 〜レールを超えたその先へ〜
kentaroutakeda
12
3.6k
AI時代のプログラミング教育 / programming education in ai era
kishida
23
21k
Kubernetesで実現できるPlatform Engineering の現在地
nwiizo
2
1.7k
いまさら聞けない生成AI入門: 「生成AIを高速キャッチアップ」
soh9834
12
3.8k
新卒から4年間、20年もののWebサービスと 向き合って学んだソフトウェア考古学
oguri
8
6.9k
英語 × の私が、生成AIの力を借りて、OSSに初コントリビュートした話
personabb
0
120
Defying Front-End Inertia: Inertia.js on Rails
skryukov
0
190
小さく段階的リリースすることで深夜メンテを回避する
mkmk884
2
130
パスキーのすべて / 20250324 iddance Lesson.5
kuralab
0
130
AI Agents with JavaScript
slobodan
0
140
PHPUnit 高速化テクニック / PHPUnit Speedup Techniques
pinkumohikan
1
1.2k
アプリを起動せずにアプリを開発して品質と生産性を上げる
ishkawa
0
400
Featured
See All Featured
Embracing the Ebb and Flow
colly
85
4.6k
A Philosophy of Restraint
colly
203
16k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
30
1.1k
Java REST API Framework Comparison - PWX 2021
mraible
29
8.5k
Optimising Largest Contentful Paint
csswizardry
35
3.2k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Writing Fast Ruby
sferik
628
61k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
2.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
29
2k
Code Reviewing Like a Champion
maltzj
522
39k
Transcript
#1'CDDʹΑΔ τϨʔγϯάೖ Ծ 2018/11/5 OSS
͜ͷൃදͷ༰ 2 BPFʹΑΔτϨʔγϯάͷ෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ͍ํͷઆ໌ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
1 Linux Tracing System 4
͡Ίʹ 5 BPFͰͷτϨʔγϯά = º શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏ͷ
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic
Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace
Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz
σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝ͰใΛऔಘ ɾ*1$ Ωϟογϡώοτ ʜ Performance Counter (PMU)
ɾ4UBUJD5SBDJOH ɾΧʔωϧʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔΛొͰ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔΛొͰ͖Δ Kprobe 1 2 3
1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ *1$ Ωϟογϡώοτ ذ༧ଌώοτ ʜ ɾ*OUFMͷ߹
ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ ɾͲͷใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷʹୡͨ͠߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸҎ্ͷใΛऔಘ͍ͨ͠߹͏·ׂ࣌͘͢Δඞཁ͕͋Δ
5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε ͍͍ͩͨ5SBDFQPJOUͷఆٛ
ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯΠϯ λϑΣʔεతͳޓੑ͕͋Δʢͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
,QSPCF 11 Insn Break point pre handler post handler Insn
( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧͷେ෦͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
ɾ-JOVYඪ४උͷϓϩϑΝΠϥ ɾΧʔωϧϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ
ɾΠϕϯτͷൃੜճͷΧϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧΛར༻ 1FSG DGQFSGGUSBDFͷΈ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
2 Tracing with BPF
5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event
Call BPF Program Helper Function pid uid … eBPF Map perf buffer
15 bpf(2) system call Create BPF map Kernel Userland BPF
map User Program
16 bpf(2) system call Verifier C source BPF Program JIT
(Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
17 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
18 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
#1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ݁ՌΛऔಘ ɾ݁ՌʹΛͯ͠NBQʹॻ͖͢ Πϕϯτൃੜճͷܭ ɾϖΞͱͳΔؔΛݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔͰ࣌ࠁΛऔಘɼNBQʹ֨ೲ
ɾFQJMPHVFͷؔͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
#1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (
( ( * , * ,( * ( ( ( *) ( * (
Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF
6QSPCF 5SBDFQPJOU 1FSGTPGUXBSF IBSEXBSFFWFOU https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
#1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
3 Tracing with bcc
#1'ϓϩάϥϜ࡞Ͱେมͳ • υΩϡϝϯτෆɼγεςϜίʔϧͷཧղ͕େม • CQG QFSG@FWFOU@PQFO ͱ͍͏ڧఢ •
$ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆελοΫʹஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞ͨ͠#1'ϓϩάϥϜ&-'όΠφϦ • &-'όΠφϦΛదʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͍ʹ͍͘
CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ τϨʔγϯάʹݶఆ͢ΔͷͰͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
1ZUIPO (P ˞τϨʔγϯάίʔυࣗମ$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
CDDͰͷϓϩάϥϜྫ 26 finish_task_switch() kprobe Python map
! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
.PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'
7+19 &… &Clang$%%)5 :%#* AST & eBPF map/8 &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
CDDDPNQJMFS 28 ) : : (
πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&
ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
πʔϧͱͯ͠ͷCDD 30
πʔϧͱͯ͠ͷCDD 31
CDDͷܽ ༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹجຊతʹҾͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ໘ଟʑ͋Δ ґଘ͕ؔ૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
Complementary 33
ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply
(Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
(P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf
loader (elfόΠφϦࣗͰίϯύΠϧ͢Δ)
ຊઆ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿΒͳͯ͘)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38