Spying Linux processes

Transcript

1. Spying Linux processes @javierhonduco
2. Before this summer
3. CODE CODE *printf debugging* CODE
4. This summer
5. CODE *printf debugging* *debugging* *wth is wrong… it _should work_!* CODE *WAT* *debugging* *moar printf debugging* CODE
6. *debugging* *WAT* *debugging* *debugging* *debugging* *eats a cookie* *debugging* CODE
7. ¯\_(ツ)_/¯
8. The OS ~= API • Network: bind(), listen(), accept(), read()... • Files: open(), read(), write()... • Memory allocation: mmap(), malloc(), brk()? • Threads et al: pthreads_
9. Mmmm, something’s not quite ok!
10. 1. My process is kindof stuck for no reason! 2. The file it should write is empty! 3. Its socket doesn’t get a single byte! 4. I just run out of FDs, but I was only using one 5. {CPU, disk, memory} usage is too high
11. Let’s peek into!
12. 1. Stuck process. Let’s strace!
13. strace -p <pid>
15. 2. It’s writing to another file! Let’s strace again
16. It opens the wrong file!!! But I set another path in the “WHATEVER_LOGGER” ENV variable...
17. Let’s into /proc/<pid>/environment
18. Ooops! The ENV var was not set in the appropriate place
19. /proc/<pid>/<*> is pretty rad Exposes kernel data structures in the VFS
20. 3. It doesn’t reach a server ngrep to the rescue!
23. 4. I run out of FDs asks coworker :D
24. “Javier, try with lsof”
25. lsof -p <pid>
26. Thousands like this!
27. It ended up being a bug on a Ruby library written in C
28. 5. CPU DISK RAM
29. perf (A bit out of the scope of this talk. Also, I’m even more newbie on this! )
30. Graphs!
32. Linux 4.1 and above… BPF Compiler Collection (BCC) Basically low overhead kernel tracing!
33. Flamegraphs, heatmaps, histograms etc etc
34. valgrind --leak-check=yes \ ./maybe_leaking_program
35. What have I learnt? • Unix tools are awesome! • /proc/<pid>/<*>!! • I write lots of bugs! (but hopefully, got a bit better at debugging) • Things are going to fail in every single way they can (and that could be fun!)
36. Merci!
37. Interesting links/ bibliography [1] iovisor: https://github.com/iovisor/bcc/ [2] Julia Evans: http://jvns.ca/ [3] perf: https://perf.wiki.kernel.org/index.php/Main_Page [4] Brendan Gregg: http://www.brendangregg.com/ [6] BPF syntax http://biot.com/capstats/bpf.html [7] Man pages are useful too! (but I do need examples as well :P)