eBPF & the future of osquery on Linux

Slides from the talk at osquery@scale 2021.

Transcript

1. eBPF and the future of osquery on Linux Zach Wasserman – CTO, Fleet osquery@scale 2021
2. Who am I • Co-creator of osquery • Cofounder & CTO of FleetDM • (Former) Cofounder & Principal Engineer of Kolide
3. Silent Accomplice Alessandro Gario Senior Software Engineer Trail of Bits
4. Audit & Osquery
5. Audit & Osquery Tables • Audit powers many of the event-based tables for osquery on Linux • process_events • process_file_events • socket_events • user_events • apparmor_events • selinux_events
6. Audit & Osquery Con f i guration • Base con f i guration to enable audit • --disable_audit=false • --audit_allow_config=true • --audit_persist=true
7. Audit & Osquery Con f i guration • Enable each feature separately • --audit_allow_apparmor_events • --audit_allow_fim_events • --audit_allow_fork_process_events • --audit_allow_kill_process_events • --audit_allow_process_events • --audit_allow_selinux_events • --audit_allow_sockets • --audit_allow_user_events
8. Audit Drawbacks
10. Audit Drawbacks There can be only one • Audit’s design allows only a single consumer of the generated events • Receiving audit events in osquery means disabling auditd • Disable auditd -> No audit events written to f i le • Some tools expect to be able to retrieve audit events from f i le! Highlander Quote - Linux Audit: Moving Beyond Kernel Namespaces to Audit Container IDs - Richard Guy Briggs (Linux Security Summit 2018)
11. Audit Drawbacks There can be only one • Many SELinux tools rely on audit logs in /var/log/audit/audit.log • sealert • audit2allow
12. Audit Drawbacks Containers • Audit does actually work with containers • Fixed in Red Hat Bug 893751 • Audit is not “aware” of containers • Lack of namespace information hinders usability in container workloads
13. eBPF
14. https://ebpf.io/what-is-ebpf eBPF Programmable Hooks
15. eBPF Safety • Required privileges • Program veri f i cation • Limited size • Limited complexity • Bounded loops • Controlled memory access
16. eBPF & Osquery
17. eBPF & Osquery State • eBPF functionality released in osquery 4.6.0 • Built primarily by Alessandro Gario • Implemented on top of github.com/trailofbits/ebpfpub
18. eBPF & Osquery Tables • bpf_process_events • bpf_socket_events • …
19. eBPF & Osquery Con f i guration • --enable_bpf_events • That’s it!
20. eBPF & Osquery Tuning • --bpf_buffer_storage_size (default 512) • --bpf_perf_event_array_exp (default 10)
21. eBPF & Osquery Compatibility • Targeting Kernels 4.18+ (2018) • Possible to extend compatibility back to 4.10+ (2017) • eBPF Probes are generated at runtime • One binary can work on most Kernels
22. eBPF & Osquery Coming soon… • Support for correlating BPF events with containers • Mapping cgroup_ids to Docker containers • process_dns_events
23. eBPF & Osquery Future • We now have a pattern for instrumenting nearly anything on Linux • System calls • Kernel tracepoints • User-space function calls • These can be dynamically con f i gured at osquery runtime
24. eBPF & Osquery Future - Security • Instrument any and all syscalls of interest • Track signals sent to processes • Kernel module loads/unloads • Track LD_PRELOAD
25. eBPF & Osquery Future - Devops/SRE • Instrument any and all syscalls of interest(!) • Measure latency and resource consumption of OS processes • Network stack • Filesystem • Other I/O • Count and measure functions within user-space
26. eBPF & Osquery Future - Imagine • Let’s look at the tools of today • bpftrace • BCC • Which of these use cases map well to osquery’s SQL model? • How can osquery be useful for shipping the aggregated information from hosts?
27. Conclusion
28. Audit & eBPF are both viable approaches
29. eBPF has potential to dramatically increase scope of observability with osquery
30. Thank you zach@fleetdm.com @thezachw @zwass