The complete story of the in-kernel sloppy GPIO logic analyzer

You know the rules, right? You should not implement a logic analyzer in software because equidistant sampling points cannot be guaranteed. You should also not disable interrupts in your kernel code, and if you really need to, it should be as short as possible. And surely you should not allow userspace to allocate memory without any constraints. Now, one big joy with hacking is breaking such rules and go wild. Listen to Wolfram’s story about the sloppy GPIO logic analyzer. How it started as an internal tool to debug a remote system and evolved into something which is making its way to mainline. You will learn details about upstreaming, the GPIO subsystem, that isolating a CPU core for one specific task is harder than it looks, and that Murphy’s law is still all around in 2022. Oh, and that Kernel hacking is tons of fun, of course.

Wolfram Sang

Transcript

1. The complete story of the in-kernel sloppy GPIO logic analyzer Wolfram Sang, Consultant / Renesas 03.06.2022, KernelRecipes Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 1 / 31
2. The elephant in the room In-kernel sloppy GPIO logic analyzer samples data by polling GPIOs on an “isolated” CPU core does so with irqs + preemption disabled Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 2 / 31
3. The elephant in the room In-kernel sloppy GPIO logic analyzer samples data by polling GPIOs on an “isolated” CPU core does so with irqs + preemption disabled Why is it called “sloppy”? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 2 / 31
4. The elephant in the room In-kernel sloppy GPIO logic analyzer samples data by polling GPIOs on an “isolated” CPU core does so with irqs + preemption disabled Why is it called “sloppy”? Not really a question, or?? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 2 / 31
5. Questions? Wait until the end, please1 1It’s a bit of a ride… Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 3 / 31
6. The Task enable IP cores on new board cores were known add dt-bindings enable clocks set pinmuxing We only sent tested patches upstream Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 4 / 31
7. The Problem Wolfram at home (.de) 2Easy for I2C, MMC, …; not so easy for PWM, … Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 5 / 31
8. The Problem Wolfram at home (.de) board in lab (.jp)2 2Easy for I2C, MMC, …; not so easy for PWM, … Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 5 / 31
9. The idea What did I not have multiple logic analyzers in the lab a person constantly in the lab Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 6 / 31
10. The idea What did I not have multiple logic analyzers in the lab a person constantly in the lab What did I have wires to be set up once lots of idle CPU cores Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 6 / 31
11. Was I an expert in CPU isolation? 3Really high risk of brown paper bag situation here Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 7 / 31
12. Was I an expert in CPU isolation? No3 3Really high risk of brown paper bag situation here Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 7 / 31
13. Did I develop a thick skin after all these years? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 8 / 31
14. Did I develop a thick skin after all these years? Maybe Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 8 / 31
15. Do I want to share what I developed? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 9 / 31
16. Do I want to share what I developed? YES! Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 9 / 31
17. SGLA: the kernel part relatively easy config files in debugfs out: freq, size, triggers, start in: data, meta data, timing info when run: lock CPU wait for triggers sample unlock CPU Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 10 / 31
18. SGLA: the script relatively complex provide easy syntax for the debugfs files “isolate” CPU run polling task on that CPU convert data to sigrok format Should run everywhere, so any shell Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 11 / 31
19. script: avoid bashism only ash, zip, and taskset needed verified by check_bashisms Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 12 / 31
20. script: avoid bashism only ash, zip, and taskset needed verified by check_bashisms verified by shellcheck Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 12 / 31
21. script: avoid bashism only ash, zip, and taskset needed verified by check_bashisms verified by shellcheck verified by Andy Shevchenko Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 12 / 31
22. script: isolate CPU I thought there was a helper for all this?4 set up a cpuset set smp_affinity for all irqs set cpumask for all workqueues use taskset to move tasks away from “isolated” CPU tell RCU that stalled CPUs are okay (set cpufreq governor to performance) 4isolcpus is nice but deprecated Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 13 / 31
23. DT example i2c2-analyzer { compatible = "gpio-sloppy-logic-analyzer"; probe-gpios = <&gpio6 21 GPIO_OPEN_DRAIN>, <&gpio6 4 GPIO_OPEN_DRAIN>; probe-names = "SCL", "SDA"; }; Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 14 / 31
24. Live demo #1 Snoop I2C traffic on a local machine Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 15 / 31
25. Result from Live Demo #1 Figure 1: Not so live Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 16 / 31
26. Commercial break: sigrok Great project… Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 17 / 31
27. Commercial break: sigrok Great project… having serious man-power problems :( Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 17 / 31
28. Live demo #2 Snoop PWM on a remote machine Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 18 / 31
29. Murphy’s law, take #1 # gpio-sloppy-logic-analyzer -s 1500000 \\ -d 12000 \\ -t "1H+2H,1H+2F" Could not isolate CPU1. Does it exist? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 19 / 31
30. Murphy’s law, take #1 # gpio-sloppy-logic-analyzer -s 1500000 \\ -d 12000 \\ -t "1H+2H,1H+2F" Could not isolate CPU1. Does it exist? Oh, no SMP yet? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 19 / 31
31. Murphy’s law, take #1 # gpio-sloppy-logic-analyzer -s 1500000 \\ -d 12000 \\ -t "1H+2H,1H+2F" Could not isolate CPU1. Does it exist? Oh, no SMP yet? What to do? Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 19 / 31
32. Result from Live Demo #2 It worked, really, trust me! Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 20 / 31
33. Let’s skip the wiring! Ulrich’s idea he read the GPIOIN reg and could skip the wiring Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 21 / 31
34. Let’s skip the wiring! Ulrich’s idea he read the GPIOIN reg and could skip the wiring I hacked the GPIO subsystem to reuse GPIOs Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 21 / 31
35. Let’s skip the wiring! Ulrich’s idea he read the GPIOIN reg and could skip the wiring I hacked the GPIO subsystem to reuse GPIOs Linus mentioned that this was a known and supported feature in the GPIO subsystem called “non-strict” Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 21 / 31
36. Let’s skip the wiring! Ulrich’s idea he read the GPIOIN reg and could skip the wiring I hacked the GPIO subsystem to reuse GPIOs Linus mentioned that this was a known and supported feature in the GPIO subsystem called “non-strict” we can measure pins which are not even exposed on the board \o/ Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 21 / 31
37. Let’s skip the wiring! Ulrich’s idea he read the GPIOIN reg and could skip the wiring I hacked the GPIO subsystem to reuse GPIOs Linus mentioned that this was a known and supported feature in the GPIO subsystem called “non-strict” we can measure pins which are not even exposed on the board \o/ Our HW supports that, but not our pinctrl driver :( Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 21 / 31
38. One line fix to the rescue! - if (!pfc->gpio) { + if (!pfc->gpio && !cfg->mux_mark) { Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 22 / 31
39. DT example 2 i2c2-analyzer { compatible = "gpio-sloppy-logic-analyzer"; probe-gpios = <&gpio5 4 0>, <&gpio5 0 0>; probe-names = "SCL", "SDA"; }; Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 23 / 31
40. Live demo #3 Snoop I2C traffic on a local machine without wires Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 24 / 31
41. Result from Live Demo #3 Figure 2: Not so live Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 25 / 31
42. Live demo #4 Snoop PWM traffic on a remote machine without wires Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 26 / 31
43. Result from Live Demo #4 Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 27 / 31
44. Result from Live Demo #4 NULL Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 27 / 31
45. Murphy’s law, take #2 HW no longer non-strict with GPIOs5 5most likely Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 28 / 31
46. current status works reasonably well for my mostly idle targets sadly our HW lost the non-strict GPIOs was already useful when debugging other issues we can go even more wild lots of fun while creating (unless other people reported back ;)) still not upstream still not a logic analyzer Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 29 / 31
47. But I got a “Quote of the Week” \o/ User: Password: Log in | Subscribe | Register Quote of the week [Posted March 31, 2021 by corbet] Okay, this one is maybe a bit brave, let's see if it is suitable for upstream. This is an in-kernel logic analyzer based on GPIO polling with local irqs disabled. — Wolfram Sang (Log in to post comments) Copyright © 2021, Eklektix, Inc. Comments and public postings are copyrighted by their creators. Linux is a registered trademark of Linus Torvalds Content ▶ Edition ▶ Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 30 / 31
48. The End Questions? Comments? (except from Marek ;)) Questions? Right here, right now… At the conference wsa@kernel.org And thanks go to Renesas for partly funding this work! Wolfram Sang, Consultant / Renesas The sloppy GPIO logic analyzer 03.06.2022, KernelRecipes 31 / 31