Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Designing a gRPC Interface for Kernel Tracing w...

Designing a gRPC Interface for Kernel Tracing with eBPF

As a maintainer of the CNCF runtime security project, Falco, he was tasked with designing a mutually TLS authenticated API over gRPC in C/C++ to solve the runtime security problem. Join this talk to understand the challenges he faced with designing the interface, as well as the performance concerns with parsing millions of syscalls using eBPF over gRPC. The audience will walk away with an understanding of runtime security in cloud-native, as well as the technical concerns with building such an interface.

KubeCon + CloudNativeCon Europe 2020

Leonardo Di Donato

August 19, 2020
Tweet

More Decks by Leonardo Di Donato

Other Decks in Technology

Transcript

  1. A timeline always works fine Falco created to parse libsinsp

    events! May 2016 Accepted as a CNCF incubation level hosted project Jan 2020 Sysdig Inc. donated Falco to the CNCF Oct 2018 2 May 2019 Falco Community Calls start! @leodido
  2. Leonardo Di Donato Open Source Software Engineer Falco Maintainer @leodido

    3 extra points to who spots the meaning of this Italian hand-gesture! Whoami!
  3. Contents 4 Intro Tech for the cool hardcore kids, yet

    not cloud-native eBPF The problem of providing runtime security by tracing the Linux kernel - ie., Falco Make eBPF maps cloud-native through gRPC gRPC 1 2 3 @leodido
  4. Security 5 Use policies to change the behavior of a

    process by preventing syscalls from succeeding (also killing the process sometimes). Detection Use policies to monitor the behavior of a process and notify when its behavior steps outside the policy. Prevention @leodido
  5. Security 6 sandboxing, access control ❏ seccomp ❏ seccomp-bpf ❏

    SELinux ❏ AppArmor ❏ Cloud-Native Security ❏ PSP ❏ policy-based admission plugins ❏ network policies ❏ ... Auditing behavioral monitoring, intrusion & anomaly detection, forensics ❏ auditd ❏ Falco ❏ ... ❏ a lot still to be done in this space! Enforcement @leodido
  6. Code (Applications) Cluster Container Cloud/Co-Lo/Corporate Data Center Prevention is not

    enough. OS Kernel Combine with runtime detection tools. Use a defense-in-depth strategy. @leodido
  7. She’s Kelly. I have a lock on my front door

    and an alarm, but she alerts me when things aren’t going right, when little bro is misbehaving or if there’s someone suspicious outside or nearby. She detects runtime anomalies in my life at home. Runtime Security Thanks @ckranz for the inspiration!
  8. “The system call is the fundamental interface between an application

    and the Linux kernel.” 9 — man syscalls 2 @leodido
  9. Syscalls only are not enough, too. ‍♂ 10 Context ❏

    timing ❏ arguments Containers ❏ Did the event originated in a container? ❏ What is the container name and ID? ❏ What is the container image? Orchestrator ❏ In which cluster it is running? ❏ On which node? ❏ What is the container runtime interface in use? @leodido
  10. Kernel module Pros: very efficient, implement almost anything Cons: kernel

    panics, not always suitable eBPF probe Pros: program the kernel without risking to break it Cons: newer kernels pdig Pros: (almost) unprivileged Cons: really hackish, ~20% slower Other methods? Future inputs/drivers? 11 How to get syscalls to userspace? @leodido
  11. eBPF Not just packet filtering anymore. You can now write

    mini programs that run on events (kernel routine execution, disk I/O, syscall) which are run in a safe register-based VM using a custom 64 bit RISC instruction set in the kernel. The In-kernel verifier refuses to load eBPF programs with: ❏ invalid or bad pointer dereferences ❏ exceeding maximum call stack ❏ loops without an upper bound ❏ ... Stable Application Binary Interface (ABI). @leodido @leodido
  12. How does eBPF work? networking load compile user-space kernel BPF

    source BPF ELF bpf() verifier BPF Maps Maps data kprobe uprobe static tracepoint perf event XDP (net driver) eBPF opcodes eBPF maps BPF_PROG_LOAD BPF_MAP_CREATE cgroups tc (traffic control) tracing/monitoring socket filter BPF_PROG_TYPE_SOCKET_FILTER BPF_PROG_TYPE_KPROBE BPF_PROG_TYPE_TRACEPOINT BPF_PROG_TYPE_RAW_TRACEPOINT BPF_PROG_TYPE_XDP BPF_PROG_TYPE_PERF_EVENT BPF_PROG_TYPE_CGROUP_SKB BPF_PROG_TYPE_CGROUP_SOCK BPF_PROG_TYPE_SOCK_OPS BPF_PROG_TYPE_SK_SKB BPF_PROG_TYPE_SK_MSG BPF_PROG_TYPE_SCHED_CLS BPF_PROG_TYPE_SCHED_ACT See enum bpf_prog_type at bit.ly/bpf_prog_types @leodido
  13. eBPF maps: sharing state between kernel and userspace async in-kernel

    key-value store Each map type has: ❏ a type ❏ a max number of elements ❏ key size (bytes) ❏ value size (bytes) map operations ❏ BPF_MAP_CREATE ❏ BPF_MAP_LOOKUP_ELEM ❏ BPF_MAP_UPDATE_ELEM ❏ BPF_MAP_DELETE_ELEM ❏ BPF_MAP_GET_NEXT_KEY ❏ ... See enum bpf_cmd at bit.ly/bpf_map_commands so many map types ❏ BPF_MAP_TYPE_HASH ❏ BPF_MAP_TYPE_ARRAY ❏ BPF_MAP_TYPE_PROG_ARRAY ❏ BPF_MAP_TYPE_PERF_EVENT_ARRAY ❏ BPF_MAP_TYPE_LPM_TRIE ❏ BPF_MAP_TYPE_PERCPU_HASH BPF_MAP_TYPE_PERCPU_ARRAY ❏ … See enum bpf_map_type at bit.ly/bpf_map_types @leodido
  14. Syscalls from Falco eBPF probe 15 kernel space user space

    libsinsp libscap eBPF VM eBPF maps eBPF probe @leodido
  15. Ready to start capturing! 1. collect machine info (# online

    cores), enable eBPF JIT, get iface, process and user list 2. parse the ELF of the eBPF object file a. check eBPF probe version matches Falco driver version b. look for “maps” sections and populate them i. SYSCALL_CODE_ROUTING_TABLE, SYSCALL_TABLE, EVENT_INFO_TABLE, FILLERS_TABLE, ... c. look for “tracepoint”, “raw_tracepoint” prefixed ELF sections i. load them: bpf() syscall (BPF_PROG_TYPE_TRACEPOINT or BPF_PROG_TYPE_RAW_TRACEPOINT) ii. attach them: open /sys/kernel/debug/tracing/events/<event>/id + ioctl(..., PERF_EVENT_IOC_SET_BPF), or bpf(BPF_RAW_TRACEPOINT_OPEN) d. “filler” prefixed ELF sections i. lookup FILLERS_TABLE and populate BPF_MAP_TYPE_PROG_ARRAY eBPF map ii. executed when corresponding syscall entry/exit (filler/<syscall-event>) get traced 3. scan “/proc” fs How actually loading an eBPF program looks like ‍♂ libscap scap_open_live_int(), scap_bpf_load(), load_bpf_file(), load_elf_maps_section(), load_maps(), load_tracepoint(), populate_*_map() @leodido
  16. ❏ Working on top of HTTP2 ❏ stream multiplexing within

    single connection, … ❏ Streaming calls ❏ client streaming, server streaming, both ❏ Implementations in many languages ❏ Authentication systems ❏ Strong protocol typing ❏ protobuf, flatbuffer, … ❏ Many rich features ❏ retries, flow control, cancellation, deadlines etc. gRPC @leodido
  17. That’s the question. Sync Async Pros ❏ simple to use

    and get going ❏ efficiency ok for most applications Cons ❏ code can be called concurrently from multiple threads (same or different clients) ❏ worker thread occupied until RPC finishes ❏ unary RPCs can block a thread for long time if handling requires blocking IO ❏ long running streaming RPCs always block a thread Pros ❏ bring your own threading model ❏ state-of-the-art at scaling ❏ best performances for who’s willing to go the extra mile Cons ❏ implementing its API needs considerable boilerplate code ❏ some implicit behaviors not called out very well in the documentation man 7 epoll @leodido
  18. outputs.proto Falco gRPC Outputs API falco.outputs.service/sub Long-lived (bidi) streaming RPC

    Get notified when some Falco rules violations happen and wait. @leodido
  19. outputs.proto Falco gRPC Outputs API falco.outputs.service/get Server streaming RPC Get

    all the Falco rules violations happened and stop. @leodido
  20. Optimize Tools + Benchmarks ❏ GRPC_TRACE ❏ gprof, pprof ❏

    valgrind, mutrace ❏ experimental interceptors ❏ application benchmarks ❏ synthetic benchmarking More at falco#1241 Suggestions ❏ Use the Async API ❏ Tune the threading model ❏ Tune the number of completion queues ❏ Reduce contention ❏ Reduce allocations ❏ Reduce copies ❏ Measure outstanding RPCs @leodido
  21. Future work ❏ multiple unary RPCs rather than long running

    bidirectional? ❏ improve the half-duplexing of the existing bidirectional outputs RPC ❏ one output queue per session/context ❏ falcosecurity/client-go, client-py, client-rs ❏ go examples, asciinema py, asciinema rs Contributors wanted! Join the Falco community and help us! @leodido
  22. Questions and feedback welcome 31 Thanks! ❏ twitter.com/leodido ❏ github.com/leodido

    ❏ github.com/falcosecurity/falco ❏ slack.k8s.io, #falco channel ❏ thanks to Apulia for inspiration