Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Diving into the Details with DTrace
Search
Colin Jones
August 18, 2016
Programming
3
490
Diving into the Details with DTrace
A talk from Abstractions 2016
Colin Jones
August 18, 2016
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
140
Underestimated costs of microservice architectures
trptcolin
3
1.5k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
390
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
340
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
160
Finding out what's *really* going on, with DTrace!
trptcolin
1
320
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
120
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
trptcolin
1
650
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
160
Other Decks in Programming
See All in Programming
Why Spring Matters to Jakarta EE - and Vice Versa
ivargrimstad
0
1k
役立つログに取り組もう
irof
27
8.7k
Hotwire or React? ~Reactの録画機能をHotwireに置き換えて得られた知見~ / hotwire_or_react
harunatsujita
9
4.1k
Kaigi on Rails 2024 - Rails APIモードのためのシンプルで効果的なCSRF対策 / kaigionrails-2024-csrf
corocn
5
3.4k
ピラミッド、アイスクリームコーン、SMURF: 自動テストの最適バランスを求めて / Pyramid Ice-Cream-Cone and SMURF
twada
PRO
9
1k
Golang と Erlang
taiyow
8
1.9k
offers_20241022_imakiire.pdf
imakurusu
2
360
OpenTelemetryでRailsのパフォーマンス分析を始めてみよう(KoR2024)
ymtdzzz
4
1.6k
カラム追加で増えるActiveRecordのメモリサイズ イメージできますか?
asayamakk
4
1.6k
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
870
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
140
Piniaの現状と今後
waka292
5
1.5k
Featured
See All Featured
What's in a price? How to price your products and services
michaelherold
243
12k
Faster Mobile Websites
deanohume
304
30k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.8k
Adopting Sorbet at Scale
ufuk
73
9k
How to Ace a Technical Interview
jacobian
275
23k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Designing for Performance
lara
604
68k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
22k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Transcript
Diving into the details with DTrace! Colin Jones @trptcolin
systems applications
problems solutions
problems solutions
What even is DTrace?
DTrace is… strace
DTrace is… strace++
strace++ $ sudo dtrace -qn ' syscall:::entry { @[probefunc, execname]
= count(); } dtrace:::END { trunc(@, 10); printa(@); }'
strace++ $ sudo dtrace -qn 'syscall:::entry { @[probefunc, execname] =
count(); } dtrace:::END { trunc(@, 10); printa(@); }' ^C read Slack 1993 select Slack 2017 kevent_qos Slack 2660 kevent_qos WindowServer 2801 psynch_cvsignal Box Sync 2873 psynch_cvwait Box Sync 2873 select Box Sync 4780 madvise Slack 5783 bsdthread_ctl Slack 6910 workq_kernreturn Slack 10622
DTrace is… strace++ system resources
system resources $ sudo dtrace -qn ' vminfo:::maj_fault { @[execname]
= sum(arg0); }'
system resources $ sudo dtrace -qn 'vminfo:::maj_fault { @[execname] =
sum(arg0); }' ^C 2BUA8C4S2C.com.a 1 mdworker 2 Google Chrome 12 mds_stores 116 Google Chrome He 252
DTrace is… strace++ system resources statically-defined tracing
statically-defined tracing $ sudo dtrace -qn ' mysql*::: { @[probename]
= count(); } dtrace:::END { trunc(@, 10); printa(@); }'
$ sudo dtrace -qn 'mysql*::: { @[probename] = count(); }
dtrace:::END { trunc(@, 10); printa(@); }' ^C query-exec-start 3 query-parse-done 3 query-parse-start 3 query-start 3 select-done 3 select-start 3 net-read-done 18 net-read-start 18 net-write-done 18 net-write-start 18 statically-defined tracing
DTrace is… strace++ system resources statically-defined tracing dynamic tracing
Dynamic tracing $ sudo dtrace -qn ' fbt:mach_kernel::entry { self->in
= timestamp; } fbt:mach_kernel::return /self->in/ { @ = quantize(timestamp - self->in) self->in = 0; }'
Dynamic tracing $ sudo dtrace -n 'fbt:mach_kernel::entry { self->in =
timestamp; } fbt:mach_kernel::return /self->in/ { @ = quantize(timestamp - self->in); self->in = 0; }' dtrace: description 'fbt:mach_kernel::entry ' matched 24482 probes ^C value ------------- Distribution ------------- count 256 | 0 512 |@@@@@@@@@@@@@@@@@@ 13521 1024 |@@@@@@@@@@@@@@@@@@@ 14961 2048 |@@@ 2172 4096 | 79 8192 | 71 16384 | 7 32768 | 3 65536 | 3 131072 | 2 262144 | 0
DTrace is… strace++ system resources statically-defined tracing dynamic tracing
Weird slowness… it happens
Slooooooooow tests (…sometimes)
Multiple tests, multiple machines
Affects some teammates worse than others
annoying productivity-sapping stressful
Hypotheses?
Use DTrace!
Rule out the usual suspect(s)? hotspot*:::gc-begin { self->start = timestamp;
} hotspot*:::gc-end /self->start/ { this->time = (timestamp - self->start) / 1000000; printf("%-8d %-8d\n", pid, this->time); }
Garbage Collection $ sudo gc_time.d PID GC (ms) 40320 9
40320 9 73113 1 73113 1 73113 1 40320 36 73184 1 73184 1 73184 1 40320 9 40320 160 72735 7
System resources CPU Memory Disk Network
CPU? $ sudo dtrace -qn ' profile-997 { @[execname] =
count(); } '
CPU? $ sudo dtrace -qn 'profile-997 { @[execname] = count();
}' ^C [...] Box Sync 3 Google Drive 3 java 3 Google Chrome 9 systemstatsd 11 iTerm 14 Google Chrome He 24 WindowServer 25 hidd 26 Box Sync Monitor 47 kernel_task 9527
CPU? $ sudo dtrace -qn ' profile-997 /execname=="kernel_task"/ { @[stack()]
= count(); } '
CPU? $ sudo dtrace -qn 'profile-997 /execname=="kernel_task"/ { @[stack()] =
count(); }' ^C [...] kernel`0xffffff80010f3d30+0x358 kernel`0xffffff800158d890+0x793 kernel`kevent+0x44 kernel`unix_syscall64+0x251 kernel`hndl_unix_scall64+0x16 116 kernel`processor_idle+0x107 121 kernel`machine_idle+0x2e0 kernel`call_continuation+0x17 82967
CPU $ sudo dtrace -qn 'profile-997 { @[execname] = count();
}' ^C [...] Box Sync 3 Google Drive 3 java 3 Google Chrome 9 systemstatsd 11 iTerm 14 Google Chrome He 24 WindowServer 25 hidd 26 Box Sync Monitor 47 kernel_task 9527
System resources CPU Memory Disk Network
Network connections https://github.com/brendangregg/DTrace-book-scripts/blob/master/Chap6/soconnect_mac.d
Network connections https://github.com/brendangregg/DTrace-book-scripts/blob/master/Chap6/soconnect_mac.d $ sudo soconnect_mac.d PID PROCESS FAM ADDRESS
PORT LAT(us) RESULT 88161 java 2 127.0.0.1 5432 144 Success 88161 java 2 127.0.0.1 5432 171 Success 88161 java 2 127.0.0.1 5432 150 Success 114 AirPlayXPCHelper 2 192.168.1.27 7000 1762 In progress 88161 java 2 127.0.0.1 5432 141 Success 88161 java 2 127.0.0.1 5432 179 Success 88161 java 2 127.0.0.1 5432 137 Success 88161 java 2 72.52.4.119 80 29977 Success 88161 java 2 72.52.4.119 80 42121 Success 88161 java 2 72.52.4.119 80 29471 Success 88161 java 2 72.52.4.119 80 29360 Success 88161 java 2 72.52.4.119 80 34731 Success 88161 java 2 72.52.4.119 80 28824 Success
DNS pid$target::getaddrinfo:entry { self->start = timestamp; self->host = copyinstr(arg0); }
pid$target::getaddrinfo:return /self->start/ { this->now = (timestamp - timezero) / 1000000; this->time = (timestamp - self->start) / 1000000; printf("%-8d %-8d %s\n", this->now, this->time, self->host); self->start = 0; self->host = 0; }
DNS $ sudo dns_latency.d -p 41161 Password: TICK ms HOST
74450 274 redclay.local 74734 0 redclay.local 105006 30075 someplace.com 145171 30075 someplace.com 145191 1 redclay.local 145192 0 redclay.local 177055 0 example.com 252314 2 redclay.local 252315 0 redclay.local 284392 0 example.com
Now we know what to fix!
What did we learn?
Specific vs. General
DTrace is… strace++ system resources statically-defined tracing dynamic tracing
You can do it!
Where can we learn more?
Brendan Gregg's blog: http://www.brendangregg.com/dtrace.html The DTrace guide http://dtrace.org/guide DTrace: Dynamic
Tracing in Oracle Solaris, Mac OS X, and FreeBSD http://amzn.to/2bqBczw Read
Scripts that ship with OS X find /usr/bin -name "*.d"
more DTrace scripts https://github.com/brendangregg/DTrace-book-scripts Examples
DTrace: not just performance performance: not just DTrace Extrapolate
Try it out!
DTrace & SIP on El Capitan: http://internals.exposed/blog/dtrace-vs-sip.html Julia Evans' strace
zine: http://jvns.ca/blog/2015/04/14/strace-zine/ Jeff Hodges "Notes on Distributed Systems for Young Bloods": http://bit.ly/2b2mxLf Brendan Gregg's blog: http://www.brendangregg.com/dtrace.html The DTrace guide: http://dtrace.org/guide DTrace book: http://amzn.to/2bqBczw DTrace Toolkit: https://github.com/opendtrace/toolkit DTrace book scripts: https://github.com/brendangregg/DTrace-book-scripts SystemTap (Linux): https://sourceware.org/systemtap/ bcc: https://github.com/iovisor/bcc Resources
Thanks! Colin Jones @trptcolin
[email protected]