Coverage for eBPF programs

eBPF is fastly becoming the first choice for implementing tracing and security-critical applications and software.

Yet, its ecosystem lacks tooling to make developers' life easier.

Join this talk to get to know bpfcov: an open-source tool I wrote that uses the LLVM pass infrastructure to instrument your eBPF programs to collect coverage data while they run in the eBPF VM in the Linux kernel.

I bet we all have heard so much about eBPF in recent years. Isn't it?

Every day we hear about a new project using some eBPF magic underneath.

eBPF programs are written in C but compiled for a specific ISA later executed by the eBPF Virtual Machine.

LLVM has a specific backend allowing us to write C and get eBPF ELF objects out.

Still, there are no tools helping developers to clearly understand which path their code took while running, which branches were uncovered, and maybe why. Even testing the eBPF programs is a pain, given that not all the types of eBPF programs are supported by BPFPROGTEST_RUN in the Linux kernel.

Yes, BTF and CO-RE are improving the situation. But, writing eBPF is still mostly about fighting against the BPF VM verifier.

Until today, there is no simple way to visualize how the flow of your eBPF program running in the kernel actually was.

That's why I sat down and wrote bpfcov. A tool to gather source-based coverage info from your eBPF programs.

During this talk, I will show the audience the secrets of the BPF target in LLVM and how I wrote an out-of-tree LLVM pass to instrument eBPF programs with counters, counters expressions, and friends.

The goal is to help eBPF developers to get to know how to use the powerful LLVM infrastructure to make the eBPF ecosystem - and their life - better.

Transcript

1. bpfcov Coverage for eBPF programs Leonardo Di Donato - 05 Feb 2022 @ FOSDEM 22 - LLVM devroom
2. whoami Leonardo Di Donato Open Source So!ware Engineer Falco Maintainer Senior eBPF Engineer @ Elastic Security @leodido
3. why? — Lot of eBPF for tracing and security applications out there — Lot of developers approaching eBPF — No simple way for them to get coverage for their eBPF code running in the Linux kernel — Test eBPF programs via BPF_PROG_TEST_RUN, but not all program types are supported — Which path my eBPF code took while running in the kernel? Which code regions or branches got evaluated and to what? — General lack of tooling in the eBPF ecosystem @leodido
4. Goal ! Gather source-based code coverage for our eBPF applications. eBPF is: — usually written in C — compiled via Clang to BPF ELF .o files — LLVM BPF target — loaded through the bpf() syscall — executed by the eBPF Virtual Machine in the Linux kernel @leodido 4/25
5. What's source-based coverage? — Line-level granularity is not enough — AST → regions, branches, ... — Better to find grasps in the code
6. Source-based code coverage1 for C programs #include <stdio.h> #include <stdint.h> void ciao() { printf("ciao\n"); } void foo() { printf("foo\n"); } int main(int argc, char **argv) { if (argc > 1) { foo(); for (int i = 0; i < 22; i++) { ciao(); } } printf("main\n"); } $ clang \ -fprofile-instr-generate \ -fcoverage-mapping \ hello.c \ -o hello $ ./hello yay $ llvm-profdata merge \ -sparse default.profraw \ -o hello.profdata $ llvm-cov show \ --show-line-counts-or-regions \ --show-branches=count \ --show-regions \ -instr-profile=hello.profdata \ hello 1 for more details visit the LLVM docs
7. Source-based coverage — Efficient and accurate — Works with the existing LLVM coverage tools — Highlights exact regions of code (line:col to line:col) that were skipped or executed — Counts how many times a condition (branches) was taken or not (see lines 16 and 23) — Tells us what was the execution path through the code
8. -fprofile-instr-generate Instruments the program functions to collect execution counts @leodido 8/25
9. @leodido 9/25
10. -fcoverage-mapping Generate coverage mappings @leodido 10/25
11. Demystifying the profraw format 1. header 2. data (__profd_* variables) 3. counters (__profc_* variables) 4. names (__llvm_prf_nm constant) @leodido 11/25
12. Demystifying the profraw header magic __llvm_coverage_ mapping[0][3] + 1 size of __llvm_prf_cnts padding before counters size of __llvm_prf_data padding after counters size of __llvm_prf_names counters delta names begin value kind last @leodido 12/25
13. Demystifying the profraw data part @leodido 13/25
14. Demystifying the profraw counters part @leodido 14/25
15. Demystifying the profraw names part @leodido 15/25
16. Patching LLVM IR for eBPF coverage How I did it // SPDX-License-Identifier: GPL-2.0-only #include "vmlinux.h" #include <asm/unistd.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_core_read.h> #include <bpf/bpf_tracing.h> char LICENSE[] SEC("license") = "GPL"; const volatile int count = 0; SEC("raw_tp/sys_enter") int BPF_PROG(hook_sys_enter) { bpf_printk("ciao0"); struct trace_event_raw_sys_enter *x = (struct trace_event_raw_sys_enter *)ctx; if (x->id != __NR_connect) { return 0; } for (int i = 1; i < count; i++) { bpf_printk("ciao%d", i); } return 0; } // SPDX-License-Identifier: GPL-2.0-only #include <asm/unistd.h> #include <bpf/bpf.h> #include "commons.c" #include "raw_enter.skel.h" ... int main(int argc, char **argv) { struct raw_enter *skel; int err; ... /* Open load and verify BPF application */ skel = raw_enter__open(); if (!skel) ... // Set the counter skel->rodata->count = 10; err = raw_enter__load(skel); if (err) ... struct trace_event_raw_sys_enter ctx = {.id = __NR_connect}; struct bpf_prog_test_run_attr tattr = { .prog_fd = bpf_program__fd(skel->progs.hook_sys_enter), .ctx_in = &ctx, .ctx_size_in = sizeof(ctx) }; err = bpf_prog_test_run_xattr(&tattr); cleanup: raw_enter__destroy(skel); return -err; } @leodido 16/25
17. LLVM pass How I did it 1. Strip the LLVM runtime profile initialization functions/ctors 2. Ensure the eBPF program is compiled with debug info 3. Fixup visibility/linkage for eBPF globals 4. Create custom eBPF sections — __llvm_covmap → .rodata.covmap — __llvm_prf_cnts → .data.profc — __llvm_prf_data → .rodata.profd — __llvm_prf_names → .rodata.profn 5. Remove the __covrec_* constant structs — Keep them only in the BPF ELF for llvm-cov — Not in the BPF ELF for loading 6. Convert the __llvm_coverage_mapping struct to: — 2 different global arrays (header + data) 7. Convert any __profd_* struct to: — 7 different global constants (ID, hash, ..., # counters, ...) 8. Annotate with the debug info all the global variables and constants 9. Keep the llvm.used in sync @leodido 17/25
18. libBPFCov.so How I did it @leodido 18/25
19. ./bpfcov run ... How I did it 1. bpfcov run - run the instrumented eBPF application 1. Detect the eBPF globals (__profc_*, __profd_*, ...) 2. Detect their custom eBPF sections — .data.profc — .rodata.profd, — .rodata.profn — .rodata.covmap 3. Pin them to the BPF FS @leodido 19/25
20. ./bpfcov gen|out ... How I did it 1. bpfcov gen - generate the profraw from eBPF pinned maps 1. Read the content of the pinned eBPF maps at: — /sys/fs/bpf/cov/<program>/{profc,profd,profn,covmap} 2. Dump it to to a valid profraw file 2. bpfcov out - output coverage reports 1. Generates profdata files from profraw files 2. Merges them into a single one 3. HTML, JSON, LCOV coverage reports @leodido 20/25
21. Usage Compilation clang -g -O2 \ -target bpf \ -D__TARGET_ARCH_x86 \ -I$(YOUR_INCLUDES) \ -fprofile-instr-generate \ -fcoverage-mapping \ -emit-llvm -S \ -c program.bpf.c \ -o program.bpf.ll opt -load-pass-plugin $(BUILD_DIR)/lib/libBPFCov.so \ -passes="bpf-cov" \ -S program.bpf.ll \ -o program.bpf.cov.ll llc -march=bpf -filetype=obj \ -o cov/program.bpf.o \ program.bpf.cov.ll opt -load $(BUILD_DIR)/lib/libBPFCov.so \ -strip-initializers-only -bpf-cov \ program.bpf.ll | \ llc -march=bpf -filetype=obj \ -o cov/program.bpf.obj Execution sudo ./bpfcov run cov/program # Wait for it to exit # Or stop it with CTRL+C sudo ./bpfcov gen --unpin cov/program ./bpfcov out \ -o awsm_report \ --format=html cov/program.profraw @leodido 21/25
22. Demo Who wanna read LLVM IR for eBPF with me? @leodido 22/25
24. Resources — Blog post: Coverage for eBPF programs — Writing an LLVM pass — The Coverage Mapping format — Dissecting the coverage mapping sample — The encoding of the coverage mapping values: LEB128 — Demystifying the profraw format — The functions writing the profraw file: lprofWriteData(), lprofWriteDataImpl() — Source code (LLVM) emitting __covrec_* constants: CodeGen/CoverageMappingGen.cpp — Calls to CoverageMappingModuleGen in LLVM: CodeGenAction::CreateASTConsumer, CodeGenModule::CodeGenModule — Kernel patch: eBPF support for global data — Kernel patch: libbpf: support global data/bss/rodata sections — libbpf: arbitrarly named .rodata.* and .data.* ELF sections — LLVM BPF target source — How LLVM processes BPF globals — Branch Coverage: Squeezing more out of LLVM Source-based Code Coverage by Alan Phipps @leodido 24/25
25. Thank you! Questions? — twitter.com/leodido — github.com/leodido — github.com/elastic/bpfcov