Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Linux Tracing Technologies for Rust

Kenta Tada
October 09, 2021

Linux Tracing Technologies for Rust

Kenta Tada

October 09, 2021

More Decks by Kenta Tada

Other Decks in Programming


  1. R&D Center, Sony Group Corporation Copyright 2021 Sony Group Corporation

    Linux Tracing Technologies for Rust Information Exchange Meeting for Container Technologies part 15 Kenta Tada
  2. Agenda ⚫ Linux Tracing Technologies ⚫ Uprobes ⚫ USDT ⚫

    Rust with Linux Tracers ⚫ Container Runtime Debug 3
  3. Summary ⚫ Data sources • Kprobes • Tracepoints • Uprobes

    • Uprobes + USDT ⚫ Tools • Perf • BCC(BPF Compiler Collection) • bpftrace • SystemTap 7
  4. Uprobes ⚫ Uprobes : User-space Probes ⚫ Uprobes helps us

    to trace applications without modifying source codes. ⚫ If you investigate the offset of the probe point, you can probe functions. ⚫ Uretprobes provides the probe point at the exit of functions. 9
  5. Uprobes interfaces ⚫ ftrace’s interface • Ex. /sys/kernel/debug/tracing/uprobe_events ⚫ perf_event_open(2)

    • perf tools • eBPF tools ⚫ uprobe_register() in the kernel • int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc); • Kernel modules can use uprobe_register() to register a probe. – Ex. https://qiita.com/kentaost/items/1c749012d21fb2c8745e 10
  6. How to use uprobe_register() 11 #define DEBUGGEE_FILE "/home/kentaost/debuggee_app" #define DEBUGGEE_FILE_OFFSET

    (0x526) … (skipped) static int __init init_uprobe_sample(void) { int ret; struct path path; ret = kern_path(DEBUGGEE_FILE, LOOKUP_FOLLOW, &path); … (skipped) debuggee_inode = igrab(path.dentry->d_inode); path_put(&path); ret = uprobe_register(debuggee_inode, DEBUGGEE_FILE_OFFSET, &uc); https://qiita.com/kentaost/items/1c749012d21fb2c8745e ⚫ Uprobes need inode and offset. • Many tools help to transform a file name into inode easily.
  7. How to probe functions ⚫ Breakpoint instruction is used to

    probe functions. ⚫ After the handlers are executed, kernel will single-step the original instruction. 12 https://dev.framing.life/tracing/uprobes-and-int3-insn/
  8. How to set up a breakpoint for Uprobes ⚫ Existing

    processes before the prove point is registered • When uprobe_register() is called, register_for_each_vma() inserts breakpoints in existing processes. ⚫ New processes after the prove point is registered • uprobe_mmap() inserts breakpoints in new processes. • Ex. mmap(2) → mmap_region() in the kernel → uprobe_mmap() in the kernel 13
  9. Use Case : SSL sniffer ⚫ Set uprobes on SSL_write()

    and SSL_read() in TLS Library. 14 https://blog.px.dev/ebpf-openssl-tracing/
  10. USDT ⚫ USDT : Userland Statically Defined Tracing ⚫ USDT

    probes provide applications with static tracing markers. ⚫ You need to add markers in the source code manually. ⚫ You need to investigate the offset of markers from .note.stapsdt section when you trace applications. • Finally, Uprobes are used to probe the probe point. 17
  11. USDT interfaces ⚫ Add a marker in your source code

    • Install a package – Ex. # apt install systemtap-sdt-dev • Write a header in the C source code – #include <sys/sdt.h> • Write a provider at the location when you want to probe – DTRACE_PROBE2(provider, name, arg1, arg2) ⚫ When the code is compiled, USDT probes will be nop. 18 [C source code] … DTRACE_PROBE2(…) Compile [ELF Binary] … nop … .note.stapsdt
  12. How to use USDT 1. When the program is loaded

    at first, the instruction is nop. 2. The tracer program reads the .note.stapsdt section of the tracee program. 3. The tracer program changes the instruction from nop to breakpoint using Uprobes. 19
  13. Example : Slow queries ⚫ Some applications have already supported

    USDT. • MySQL, PostgreSQL, Node.js and so on. ⚫ dbslower.py from BCC is a tool to investigate slow queries. • query__start • query__done ⚫ dbslower.py tries to use Uprobes only if MySQL is without USDT support. • https://github.com/iovisor/bcc/pull/1239 20
  14. Mangling ⚫ Rust symbols are mangled in the binary except

    using the no_mangle attribute. ⚫ You need to demangle symbols to find the offset of the specified function before using Uprobes. ⚫ Tools • objdump -Ct • https://github.com/rust-lang/rustc-demangle 23
  15. USDT for Rust ⚫ All you need is to inject

    USDT probes into your ELF binary. ⚫ Some tools have been already provided. • Ex. https://github.com/cuviper/rust-libprobe 24
  16. Container Runtime is complex 26 User space Kernel space cgroup

    Low-Level Container Runtime systemd cgroupfs driver systemd driver D-Bus High-Level Container RUntime Kubelet cgroupfs driver systemd driver D-Bus
  17. Key takeaways ⚫ Uprobes is a great technology to observe

    applications. ⚫ USDT enhance the traceability of Uprobes. ⚫ Make Rust libraries more traceable using kernel technologies. 28
  18. SONY is a registered trademark of Sony Group Corporation. Names

    of Sony products and services are the registered trademarks and/or trademarks of Sony Group Corporation or its Group companies. Other company names and product names are registered trademarks and/or trademarks of the respective companies.