Test-driven kernel releases

Upstream Linux kernel testing has grown exponentially on many fronts during the past few years: kselftest is now more stable, KUnit gaining coverage and many out-of-tree test suites have kept growing. Many automated systems are running those tests continuously and regzbot has now become a central place for tracking regressions with weekly reports for mainline.

Despite these monumental achievements, mainline and stable releases still happen entirely at their maintainers’ discretion, in the absence of any known blocking regression. Linux kernel development has worked for many years while relying on subsystem maintainers’ testing tools and best-effort test reports. But in the same way that open-source has brought contributors together with a single code base, testing can bring kernel users together. Rather than having to rely heavily on downstream testing, we could bring some of the real-world quality control upstream too.

The aim of this talk is to raise a number of critical questions: What would it take to gate releases on a set of passing test results, even basic ones to start with? Can the upstream kernel community ever make such a culture shift? Could release tags include certified test results as a meaningful quality measurement for its users?

Guillaume Tucker, Collabora

Transcript

1. 1 Test-driven kernel releases Guillaume Tucker gtucker@collabora.com 2022-06-01
2. 2 Development & Testing kernel development automated testing manual testing
3. 3 Automated Testing
4. 4 Open Source Philosophy Single mainline code base Many contributors Many use-cases Application changes are sent upstream Reduced duplication of efforts mainline
5. 5 Open Testing Philosophy Single mainline code base including tests Many contributors who run tests Test results sent upstream Test results summary in each release Reduced duplication of testing efforts mainline
6. 6 Hidden Mass of Testing Hidden Mass of Testing Duplicated testing efforts Duplicated testing efforts No solution for tracking results upstream No solution for tracking results upstream Testing stays hidden as if it was downstream Testing stays hidden as if it was downstream
7. 7 automated testing
8. 8 syzbot https://syzkaller.appspot.com/ syscall fuzzing Automated bisection Reproducers Web UI
9. 9 KernelCI https://linux.kernelci.org/job/ Tailored CI system Web API Distributed test labs Kubernetes Automated bisection KCIDB database
10. 10 Red Hat CKI https://datawarehouse.cki-project.org/ Fedora kernels Mainline kernels Stable LTP, kselftest etc. KCIDB integration
11. 11 regzbot https://linux-regtracking.leemhuis.info/regzbot/mainline/ All known regressions Essentially manual submissions Seamless integration with emails Weekly report on LKML for mainline
12. 12 show me the results
13. 13 Focusing on the results Manual runs Maintainer scripts Automated systems
14. 14 Focusing on the results Manual runs Maintainer scripts Automated systems Results are the least common denominator
15. 15 Benefits of results in releases Valuable for users in general Canonical way to keep track of code quality Essentially, avoiding the “works for me” syndrome
16. 16 Challenges Shift in workflow: results are needed before the release Similar to how -rc works for stable and mainline Expect positive results rather than solely look for regressions Additional step for maintainers Keeping it simple and not disruptive Optional Up to each maintainer to decide which results to include
17. 17 in practice
18. 18 Where to start? Results reproducible on any hardware Tests included in the kernel source tree Plain builds with reference toolchain binaries, Docker images Builds with sparse enabled make C=1 coccicheck KUnit Device tree validation
19. 19 Where to start? Results reproducible on any hardware Tests included in the kernel source tree Plain builds with reference toolchain binaries, Docker images Builds with sparse enabled make C=1 coccicheck KUnit Device tree validation Documentation: https://docs.kernel.org/
20. 20 RFC 1: Test results in-tree Similar to linux-next merge logs Updated for each release (stable, mainline, -next) Rely on Git history for older results Results ├── kselftest │ ├── futex │ └── lkdtm ├── KUnit │ └── results.json └── summary
21. 21 RFC 2: Test-link in commit
22. 22 RFC 3: Git meta-data Tied to Git history Separate from commit merge workflow Similar to Git notes git results show REVISION
23. 23 Some thoughts Subsystem-specific results in separate location? Integration results for mainline / stable / linux-next Subsystem results could be pulled in alongside code Follow regular email workflow for adding results Keep in-tree result summaries in plain text Extra data can be hosted on separate systems
24. 24 RFC: <your idea here> How does the concept sound? Has this been tried or discussed before? Does it seem worth the effort? Time for an RFC on LKML to go through some details?
25. 25 Thank you!