Make Linux developers fix your kernel bug

Pssst! Don’t tell anyone, but there is a rule that forces Linux developers to fix certain kind of kernel bugs. This is pretty strictly enforced by Linux creator and lead-developer Linus Torvalds himself, who occasionally even steps up to fix such problems himself. You can benefit from this, but for that you have to do some diagnostics and report your issue properly.

This talk will explain what you need to do and what kind of problems the aforementioned rule covers. Known this will even help you when dealing with other kind of bugs: a proper report will then dramatically increase the chance that someone takes a closer look and provides a fix. To round things up the talk will also explain how to spot problems that are unlikely to get fixed, to save you from wasting time.

Thorsten Leemhuis

Transcript

1. Thorsten Leemhuis Make Linux developers fix your kernel bug
2. intro; sometimes reports on kernel bugs will just fizzle out 😕
3. intro; in rare cases, developers will be unable to fix an issues 😓
4. intro; kernel contains code nobody is really responsible for 😓
5. intro; the Linux kernel is made by volunteers
6. intro; you can't really force volunteers to do work they can't do or don't want to do
9. intro; Linux kernel developers are obliged to fix some issues! 😃
11. intro; developers will gladly address most issues in their code, 😃 if you write a decent report!if
12. intro; developers will gladly address most issues in their code, 😃 unless life gets in the way :-/
13. intro; then bad bug reports are the first developers will let fall through the cracks!
14. intro; developers will gladly address most issues in their code, 😃 unless life gets in the way :-/
15. intro; developers will gladly address most issues in their code, 😃 if you write a decent report!
16. intro; developers will gladly address most issues in their code, 😃 if you write a decent report!
17. intro; that's how you make most developers fix your bug, if they are able to
18. intro; you'll also learn when you can insist on a fix
19. intro; and how to spot issues unlikely to be fixed
20. [ act 1 ]
21. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately
27. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
28. decent report; vanilla; most kernels used in the wild are not vanilla often heavily modified & enhanced
29. decent report; vanilla; makes most distro kernel's unsuitable for reporting issues Linux kernel devs.
30. decent report; vanilla; you might want to report the issue to your Linux distributor
31. decent report; vanilla; or install a vanilla kernel yourself instead – for example a pre-built one
32. decent report; vanilla; or compile a kernel yourself hint: `make olddefconfig localmodconfig` makes things easier and relatively fast
33. decent report; vanilla; check if issue happens with a vanilla kernel, too
34. decent report; vanilla; focus on this kernel in your report, forget the distro's mentioning the distro's even briefly often just complicates report unnecessarily
35. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
37. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
38. decent report; freshness; test with latest mainline (aka -RC) release
42. decent report; freshness; focus your report on the freshest kernel you tested mentioning older briefly somewhere can be okay, but often just make report hard to grasp
44. decent report; freshness; some bugfixes are never backported to stable/longterm kernel series
45. decent report; freshness; makes longterm (LTS) kernels quite unsuitable for reporting
46. decent report; freshness; exception: regressions within a stable or longterm series something breaks 5.15.10 -> 5.15.11
47. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
49. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
51. decent report; integrity;
52. decent report; integrity;
53. decent report; integrity; Nvidia's proprietary graphics driver
54. decent report; integrity; all out-of-tree drivers are a problem incl. Nvidia's new open kernel driver
55. decent report; integrity; deinstall such drivers, reboot, check if issue still present and recheck the tainted flag!
56. decent report; integrity; many other incidents can taint kernel
57. decent report; integrity; an "Oops" for example
60. decent report; integrity; tainted kernels most of the time unsuitable for reporting bugs
61. decent report; integrity; big exception: the first Oops, warning, etc.
62. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
63. https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
64. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity [continued] d) depict the problem adequately decent report; integrity;
65. decent report; integrity; is your hardware working reliably and as specified? memtest: great idea! overclocking: stupid idea!
66. decent report; integrity; issue with file-system? fsck the volume!
67. decent report; integrity; check `dmesg -H` look out for anything red or bold
69. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
71. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
76. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS
77. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS
78. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS
79. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS
80. decent report; right place; sadly MAINTAINERS contains more than 2000 entries:-/
81. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
83. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
84. decent report; depiction; "how to write a good report" worth its own, quite long talk
85. decent report; depiction; a balancing act
86. decent report; depiction; think of it as asking for a favor
87. decent report; depiction; a favor from someone that doesn't have to help you
88. decent report; depiction; a favor from a someone that might be stressed or really short on time
89. decent report; depiction; hence, make your depiction easy to grasp for recipients
90. decent report; depiction; describe the problem neither to brief nor as a novella
91. decent report; depiction; mention version, vanilla, and taint status
92. decent report; depiction; upload & link clearly relevant logs or attach them but *don't* overload the report!
93. decent report; depiction; often relevant: output from `dmesg` & `lspci -nn`; maybe kernel's '.config', too
94. decent report; depiction; add two or three sentences summarizing the situation on top of your depiction
95. decent report; depiction; use a even more condensed and crystal-clear depiction as subject
96. decent report; depiction; in general: don't over-think or overdo your report!
97. decent report; depiction; short report will often do getting the basics right (vanilla, fresh version, no taint, easy to grasp, ...) is important
98. decent report; depiction; check for existing reports about the problem to join check <insert favorite search engine>, lore.kernel.lorg/all/, and bugzilla.kernel.org
99. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
100. decent report; depiction; it tells you to check what kind of issue you deal with
101. [ act 2 ]
102. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions
103. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue;
104. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix;
105. https://www.kernel.org/doc/html/latest/admin-guide/security-bugs.html
106. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix;
107. kind of issue; mustfix; devastating; something really really bad data is lost or damaged, hardware is bricked, ...
108. kind of issue; mustfix; devastating; make impact & urgency obvious in your report
110. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix;
111. kind of issue; mustfix; regressions; something breaks when updating the kernel say from 5.15 -> 5.16 or from 5.17.3 -> 5.17.4
112. kind of issue; mustfix; regressions; first rule of Linux kernel development: "we don't cause regressions"
114. https://linux-regtracking.leemhuis.info/about/
117. https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html
118. kind of issue; mustfix; regressions; make it obvious your report is about a regression
119. kind of issue; mustfix; regressions; CC for forward the report to regressions@lists.linux.dev
120. kind of issue; mustfix; regressions; fine print(1): only userland interfaces matter [it's thus not a regression if your out-of-tree kernel module breaks]
121. kind of issue; mustfix; regressions; fine print(2): the build config of the newer kernel version must be similar to the older one
122. kind of issue; mustfix; regressions; fine print(3): you often will be asked to find the culprit yourself
123. kind of issue; mustfix; regressions; if you find the culprit, a fix is pretty much guaranteed
124. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix;
127. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored kind of issue;
128. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits kind of issue; unlikely;
129. kind of issue; unlikely; deficits; Linux contains many incomplete drivers
130. kind of issue; unlikely; deficits; might lack a volunteer with enough time and/or motivation to improve it
131. kind of issue; unlikely; deficits; or some real-world issue prevents improvements
132. kind of issue; unlikely; deficits; check internet and docs for known deficits
133. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits kind of issue; unlikely;
134. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits II. code without an active maintainer kind of issue; unlikely;
135. kind of issue; unlikely; w/o maintainer; code often remains, as it useful for people
136. https://www.kernel.org/doc/html/latest/process/maintainers.html
137. kind of issue; unlikely; w/o maintainer; sending at least a quick brief report definitely a good idea
138. https://www.kernel.org/doc/html/latest/process/maintainers.html
139. kind of issue; unlikely; w/o maintainer; sending at least a quick brief report likely worth it
140. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored
141. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored c) all the other issues
142. kind of issue; unlikely; the rest; the quality of your report!
143. [ grand finale ]
144. take this with you
145. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
146. takeaways; almost all kernel developer are volunteers
147. takeaways; they should act on every bug report, but can and will ignore bad reports
148. takeaways; act accordingly and sent a decent report, then you'll be heard
149. takeaways; (1) check what kind of issue you deal with, as it…
150. takeaways; (a) might save you from wasting time on reporting known deficits
151. takeaways; (b) tells you what to expect from developers
152. takeaways; (2) do your homework
153. takeaways; (a) test and report with a *vanilla* kernel
154. takeaways; (b) test with a fresh mainline kernel
155. takeaways; (c) rule out local interferences
156. takeaways; (d) check MAINTAINERS to submit the report to the right place
157. takeaways; (e) write a friendly and decent report easy to gasp for others
158. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
159. takeaways; chances then are pretty good someone will help you
160. takeaways; and nearly perfect, if you report a bisected regression
161. takeaways; that's how you make the Linux developers fix kernel bugs they are able to fix
163. questions?
164. Thorsten Leemhuis mail: linux@leemhuis.info GPG Key: 0x72B6E6EF4C583D2D social media: @kernellogger, @knurd42, @knurd42rhfc, @thleemhuis and @thleemhuisfoss on #twitter & #friendica #EOF
165. Thorsten Leemhuis Make Linux developers fix your kernel bug * let me start by being fully honest * the title promises a little more than the reality can fulfill [simple reason]
166. intro; sometimes reports on kernel bugs will just fizzle out 😕 * this will always happen [not even worsed news yet]
167. intro; in rare cases, developers will be unable to fix an issues 😓 [one more bad thing]
168. intro; kernel contains code nobody is really responsible for 😓 [there is a simple reason for these three aspects]
169. intro; the Linux kernel is made by volunteers * note: volunteers != hobbyists * some of them hobbyists * most of them employees * but employees from companies contributing voluntarily [thing with them is]
170. intro; you can't really force volunteers to do work they can't do or don't want to do * motivate them * which Linus Torvalds actually does * only works up to a point * risk alienating them * might make them stop contributing * companies might decide to team up and fork [analogy helps understanding this situation]
171. * Linux is a bit like an playground built and maintained completely by a volunteers * some of those volunteers are hobbyist that wanted to build something for their kids, to learn new stuff, or enjoy helping * many of those volunteers are actually employees from local or international companies that see some benefit in helping – for example if they have a coffee or gift shop nearby [but the thing is: sooner or later all all hobbyists and companies move to something new, as their interest and priorities change over time]
172. * say because kids became adults or companies closed * some volunteer then vanish * others still help when at least kindly asked * often some other volunteer will step in * but you can force them * luckily their often is no need to * unless some play structure breaks or is found to be dangerous [and that's the same with Linux and the reason]
173. intro; Linux kernel developers are obliged to fix some issues! 😃 * if they don't they will be looked at like this
174. * luckily things seldom breaks or become dangerous, as Software doesn't decay like play structures on a playground, ;-) [more good news: developer should fix other issues as well; and most]
175. intro; developers will gladly address most issues in their code, 😃 if you write a decent report!if * as most feel proud of what they build and want to ensure it works well * thing is:
176. intro; developers will gladly address most issues in their code, 😃 unless life gets in the way :-/ * the particular developer might be short on time * stressed, sick, overwhelmed with reports or the boss forces the developer to focus on other things * thing is: that happens frequently * I guess you all known these from your life [and then…]
177. intro; then bad bug reports are the first developers will let fall through the cracks! * happens quite often * most kernel devs have a lot on their plate already * it's in your hand to prevent this fate [so let's reframe...]
178. intro; developers will gladly address most issues in their code, 😃 unless life gets in the way :-/ [as it's more like this]
179. intro; developers will gladly address most issues in their code, 😃 if you write a decent report! [emphasis]
180. intro; developers will gladly address most issues in their code, 😃 if you write a decent report! [that's why I'll tell you how to write one]
181. intro; that's how you make most developers fix your bug, if they are able to [in addition …]
182. intro; you'll also learn when you can insist on a fix * in case such a report isn't acted upon
183. intro; and how to spot issues unlikely to be fixed * safe yourself trouble [which concludes the intro and brings us…]
184. [ act 1 ]
185. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately
186. * let me please stretch the playground analogy a bit further * for two reasons [first: the example is to small; think of something bigger]
187. * these days Linux more like a really big amusement park * even bigger than this * let's call it "Linus land" * no entry free * doesn't need staff * built, maintained, and constantly improved by volunteers * that's more accurate [the second: the kernel is immaterial ]
188. * Linux is this more like an freely available ebook with instructions how to build your own "Linus land" * maintained by volunteers * and everyone everywhere can put the book into a gigantic 3d printer to build their own "Linus land" within a few minutes * or update theirs in a few minutes * sorry, a bit fat fetched, but good real life analogies for Software are hard to come by [got that? okay]
189. * you visit some park build from the book * your kid is injured on an water coaster a really good friend from school days designed * you tell your friend, who's living 2000 km away and just got a kid * friend checks the instructions for hours * can't find anything and reluctantly flies over, * notices that a few things look slightly different
190. * turns out: the one that built that park modified the book before building the park * bigger water pipes and higher water pressure were used to "improve the performance", which lead to the accident * your friend travels home really annoyed, he wasted money and hours and was blamed for an accident that's someone else fault * you don't want to do that to a friend, don't you? * that's way you don't want to do that to the volunteers that ,make Linux [and that's why want to…]
191. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; * vanilla == built from sources as distributed by kernel.org [thing is: most kernels...]
192. decent report; vanilla; most kernels used in the wild are not vanilla often heavily modified & enhanced * especially those from RHEL, SLE, and Ubuntu kernels [ such modification make... ]
193. decent report; vanilla; makes most distro kernel's unsuitable for reporting issues Linux kernel devs. * most kernel devs don't care at all about bugs with them * small mods can have a big impact * that's why some devs even reject bugs from distros that use a lightly patched kernel, like Fedora
194. decent report; vanilla; you might want to report the issue to your Linux distributor * warning: but most of the time it will be a dead end, as they don't have the resources to deal with all the reports they got [that's why you might…]
195. decent report; vanilla; or install a vanilla kernel yourself instead – for example a pre-built one * pretty easy * available for all the big distros * and a few actually use them directly [there is another option]
196. decent report; vanilla; or compile a kernel yourself hint: `make olddefconfig localmodconfig` makes things easier and relatively fast * lots of howtos on the net * use those with the mentioned commands [after installing vanilla…]
197. decent report; vanilla; check if issue happens with a vanilla kernel, too
198. decent report; vanilla; focus on this kernel in your report, forget the distro's mentioning the distro's even briefly often just complicates report unnecessarily
199. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; * concludes this point * next one is related
200. * you build a park and complain to your friend about a problem with an attraction designed by the friend * checks unsuccessfully and flies over * turns out: you used a two year old book that had a bug eliminated 18 months ago * friend was not aware of the bug as it was caused by the infrastructure used by friend's attraction * or accidentally fixed it with a big redesign or improvement * or fixed it and forgot about it [you don't want to do this to a friend, don't you?]
201. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; * kernel changes a lot all the time * bug might be fixed already [what qualifies as fresh?]
202. decent report; freshness; test with latest mainline (aka -RC) release * every bugfix has to land here first * no, it's testing RCs is dangerous, they are pretty stable * and you have backups, don't you? [find it on kernel.org]
203. * ignore the big yellow field * look at the top of table * pick that version [unless it looks like]
204. * then test the latest stable release [but when a rc-kernel exisits…]
205. * you better avoid the latest stable * developer will wonder if the bug was fixed already by someone * already increases the chances your report might be ignored * while it's not ideal to use such kernel, but not totally bad * okay as fallback * definitely better reporting with this than not at all
206. decent report; freshness; focus your report on the freshest kernel you tested mentioning older briefly somewhere can be okay, but often just make report hard to grasp [one more thing: don't use a longterm kernel]
207. * not even when a new version was released today [because]
208. decent report; freshness; some bugfixes are never backported to stable/longterm kernel series * sometimes that's simply to risky * quite a few known bugs there
209. decent report; freshness; makes longterm (LTS) kernels quite unsuitable for reporting * still better than not reporting at all * but there is a high risk that your report will not lead to anything * depends on the developer [no rule without...]
210. decent report; freshness; exception: regressions within a stable or longterm series something breaks 5.15.10 -> 5.15.11 * then it's okay to test the latest version from that series
211. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; [another important aspect follows]
212. * accidents regularly in your own up2date Linus land * say water and roller coasters stop somewhere along the track often * friend can't explain things and flies over * spot a mobile attraction in a corner of your park you allowed to come by every day and use the park's infra * friend notices the workers of the mobile attraction even modified some water pipes in the park * friend suspects the power grid is able to handle the extra load * but is not allowed to look closer at the mobile attraction, as owners consider it their "trade secret" [you don't want to annoy friends like that]
213. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; * IOW: make sure kernel remains vanilla when used and is healthy
214. * kernel can detect some integrity problems itself [check…]
215. decent report; integrity; * this is how is should look like [and not like...]
216. decent report; integrity; * everything other than a 0: * kernel likely unsuitable for reporting [one popular thing that can cause a "1" here]
217. decent report; integrity; Nvidia's proprietary graphics driver * uses the kernel in unexpected ways and even changes it * that's why most kernel developers don't care about reports with kernels using this drivers [but the thing is]
218. decent report; integrity; all out-of-tree drivers are a problem incl. Nvidia's new open kernel driver * kernel not vanilla anymore * taint number for FOSS drivers just different [that's why you...]
219. decent report; integrity; deinstall such drivers, reboot, check if issue still present and recheck the tainted flag! * then you are free to proceed with reporting [but note, there are]
220. decent report; integrity; many other incidents can taint kernel
221. decent report; integrity; an "Oops" for example * Oops = a critical error that could was detected, intercepted, and contained * kernel can continue, but is an undefined state * which can lead to subsequent faults * and thus considered unreliable * taint flag indicates that
222. [BTW: the oops shows if kernel is tainted]
223. [in the end: tainted kernels in the end…]
224. decent report; integrity; tainted kernels most of the time unsuitable for reporting bugs [exception]
225. decent report; integrity; big exception: the first Oops, warning, etc. [the kernel's docs explain this in more detail]
226. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html * reporting issues * section * links to a page dedicated to tainted kernels * which has a script and a table to decode the taint number/flags
227. https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html [there is more about integrity]
228. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity [continued] d) depict the problem adequately decent report; integrity; * there are things the kernel can't detect * that's why you better want to think about a few other things as well
229. decent report; integrity; is your hardware working reliably and as specified? memtest: great idea! overclocking: stupid idea!
230. decent report; integrity; issue with file-system? fsck the volume!
231. decent report; integrity; check `dmesg -H` look out for anything red or bold [looks like this]
232. * it might tell you what's wrong * might give you a error msg to google * and save everyone a lot of time
233. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
234. * you have a valid problem but only mention it on a school reunion where the friend later got pretty drunk and headed off with love interest from the school days * or you reported it via a chat, message board, or forum on a website you know your friend used to visit when you both where young * even after months or years, your friend didn't do anything to fix the problem * that's your fault, as you friend might not visit the website anymore * and maybe someone else is responsible anwayway these days
235. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; * web-forums definitely won't work * distro's bug tracker often a dead end as well [sadly. most of the time bugzilla.kernel.org is the wrong place, too]
236. * might looks like the central bug tracker
237. * but it's not, which you lean when you follow that link
238. bugzilla situation: it's complicated * set up by some people that thought it was a good idea * some devs liked it and started using it * but many (most?) devs never liked the idea * didn't really fit into thee email based work-flow * the idea was to have volunteers as go-between for such subsystems/maintainers * that never really worked out * that's why even today a lot of reports never reach the responsible developers (and are thus ignored) * that's why bugzilla.kernel.org often is a bad idea [instead do, what...]
239. * used by developers to find each others contact addresses
240. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS [most of the entries]
241. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS * mention the maintainer and the list * just sent your report by mail there * always CC the lists! * most prefer this way and it should always work * some subsystems uses a bug-tracker * MAINTAINERS file mentions those few that do
242. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS * mainly ACPI, PCI, and PM * about 20 out of more than 2000 entries [there are also a few that use other bug-trackers]
243. https://www.kernel.org/doc/html/latest/process/maintainers.html https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS * graphics drivers for AMD, Intel, etc.
244. decent report; right place; sadly MAINTAINERS contains more than 2000 entries:-/ * why are things so complicated and bugzilla.kernel.org? * not design, just happened over time * and no volunteer in sight to bring order into this
245. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report; [brings us to the last point]
246. * imagine your friend showing you a bug report from someone you both both went to school with, but never really liked * you read subject and first para of a report and don't get the slightest idea what this is all about * and the whole text is confusing and full of unnecessary or distracting details * and has five attachments and ten links * and is written in a unfriendly, demanding, and bearish way * reminder, friend can ignore this without consequences * would you suggest to do that?
247. 1. create a decent report a) ensure your kernel is vanilla b) ensure your kernel is fresh c) ensure your kernel's and system's integrity d) submit your report to the right place e) depict the problem adequately decent report;
248. decent report; depiction; "how to write a good report" worth its own, quite long talk
249. decent report; depiction; a balancing act
250. decent report; depiction; think of it as asking for a favor
251. decent report; depiction; a favor from someone that doesn't have to help you
252. decent report; depiction; a favor from a someone that might be stressed or really short on time
253. decent report; depiction; hence, make your depiction easy to grasp for recipients
254. decent report; depiction; describe the problem neither to brief nor as a novella
255. decent report; depiction; mention version, vanilla, and taint status * avoids doubts * mention environment (distro, hw if relevant)
256. decent report; depiction; upload & link clearly relevant logs or attach them but *don't* overload the report! * if something missing, developer will ask for it
257. decent report; depiction; often relevant: output from `dmesg` & `lspci -nn`; maybe kernel's '.config', too * depends on the issue [then]
258. decent report; depiction; add two or three sentences summarizing the situation on top of your depiction * really important to get right * developers get a lot of reports * many will stop reading after first para [some don't even get that far]
259. decent report; depiction; use a even more condensed and crystal-clear depiction as subject * also really important to get right * most people will only read that
260. decent report; depiction; in general: don't over-think or overdo your report!
261. decent report; depiction; short report will often do getting the basics right (vanilla, fresh version, no taint, easy to grasp, ...) is important [ohh, and remember]
262. decent report; depiction; check for existing reports about the problem to join check <insert favorite search engine>, lore.kernel.lorg/all/, and bugzilla.kernel.org * You might wonder: * shouldn't I have done this earlier * correct, that's why you should do things in the order described by this talk [instead]
263. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html * tries to catch local problems early * reference section providing details when you need them [document also tells you]
264. decent report; depiction; it tells you to check what kind of issue you deal with * a some require a few additional steps * there is another reason why you want to do that * it determines what you can expect * which is kinda important, too [which brings us to]
265. [ act 2 ]
266. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions [one kind are those]
267. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; * and there are three of those [the first]
268. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix; * will only happen to a few of you * but if follow the reporting issue [which will point you]
269. https://www.kernel.org/doc/html/latest/admin-guide/security-bugs.html [the second]
270. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix; * not "the paint is off" somewhere [something…]
271. kind of issue; mustfix; devastating; something really really bad data is lost or damaged, hardware is bricked, ... [luckily even more rare]
272. kind of issue; mustfix; devastating; make impact & urgency obvious in your report [and in case it's not quickly acted upon, get Linus in the loop]
273. [which brings us to the third, more common type]
274. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix; [a regression is...]
275. kind of issue; mustfix; regressions; something breaks when updating the kernel say from 5.15 -> 5.16 or from 5.17.3 -> 5.17.4 [not allowed in Linux]
276. kind of issue; mustfix; regressions; first rule of Linux kernel development: "we don't cause regressions" * coined and enforced by Linus * wants to take the fear out of updating * they nevertheless happen frequently :-/ * sadly some of the reports even fall through the cracks :-/ * that why I volunteered as the kernel's regression tracker [and built a bot]
277. * ugly, isn't it? :-/ [funding from EU]
278. https://linux-regtracking.leemhuis.info/about/ * many thx for that * project ended
280. * could talk about regressions and tracking them for hours * no time for it [once again there is a document]
281. https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html * new in 5.18 * mentioning everything important [that among others includes]
282. kind of issue; mustfix; regressions; make it obvious your report is about a regression
283. kind of issue; mustfix; regressions; CC for forward the report to regressions@lists.linux.dev [note, there is some…]
284. kind of issue; mustfix; regressions; fine print(1): only userland interfaces matter [it's thus not a regression if your out-of-tree kernel module breaks] * these modules use kernel-internal interfaces
285. kind of issue; mustfix; regressions; fine print(2): the build config of the newer kernel version must be similar to the older one * otherwise optional new features might interfere * say a new security technique blocking something a few very rare apps need * the doc I mentioned explains you how to realize
286. kind of issue; mustfix; regressions; fine print(3): you often will be asked to find the culprit yourself * many bug only happen in a certain environment * that why the change that causes often needs to be found by the reporter * the aforementioned doc explains you how to do that with a bisection using "git bisec" * sounds hard, but might only take an hour or two * initial report without this is okay, as problem might be known already [and the good thing is]
287. kind of issue; mustfix; regressions; if you find the culprit, a fix is pretty much guaranteed * and the responsible volunteer and subsystem will be known * might be possible to revert it * tell me if that doesn't work out * that's why you really want to do that in case you face a regression!
288. 2. the kind of issue at hand a) issues someone is obliged to address I. security vulnerabilities II. devastating bugs III. regressions kind of issue; mustfix; * enough about regressions and issues that have to be fixed * just one more thing you might be wondering about * who will take care of fixing such bugs * for regressions it's the author of the culprit * if MIA and for everything else it's the [maintainer]
289. [and sometimes this person]
291. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored kind of issue;
292. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits kind of issue; unlikely;
293. kind of issue; unlikely; deficits; Linux contains many incomplete drivers * a basic, incomplete driver is way better than none at all [sometimes these drivers are never improved, if…]
294. kind of issue; unlikely; deficits; might lack a volunteer with enough time and/or motivation to improve it [second reason for known deficits]
295. kind of issue; unlikely; deficits; or some real-world issue prevents improvements * example: Nouveau * docs scarce * firmware prevents using the full capabilities of the hardware * what do these known deficits mean for your report? [if it looks like a missing feature]
296. kind of issue; unlikely; deficits; check internet and docs for known deficits * prevents wasting your time on preparing a report * if in a doubt, send a quick "is this known" before writing a proper and lengthy report
297. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits kind of issue; unlikely; [another reason why some bugs are ignored]
298. 2. the kind of issue at hand b) issues most likely to be ignored I. known deficits II. code without an active maintainer kind of issue; unlikely; * Linux contains quite a bit of such code [and it remains]
299. kind of issue; unlikely; w/o maintainer; code often remains, as it useful for people * removing it would cause a regression, too * "no regression rule" should ensure it nothings break * if people like you and me tests and reports problems [two different kinds of unmaintained code]
300. https://www.kernel.org/doc/html/latest/process/maintainers.html * nearly orphaned, but not fully
301. kind of issue; unlikely; w/o maintainer; sending at least a quick brief report definitely a good idea * the "odd fixer" or someone else might take care of it [there is also fully orphaned code]
302. https://www.kernel.org/doc/html/latest/process/maintainers.html
303. kind of issue; unlikely; w/o maintainer; sending at least a quick brief report likely worth it * maybe you find others affected and can team up with them
304. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored * this concludes this section [leaves the big and wibbly-wobbly area in between those two extremes]
305. 2. the kind of issue at hand a) issues someone is obliged to address b) issues most likely to be ignored c) all the other issues [what matters here is quickly explained, as we discussed this in act 1 already]
306. kind of issue; unlikely; the rest; the quality of your report! [which brings us…]
307. [ grand finale ] * summary
308. take this with you
309. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html * looks a bit scary, but it is not * tries to catch local problems early * it's in your own interest to follow the steps [to understand why things are as they are, always keep in mind]
310. takeaways; almost all kernel developer are volunteers
311. takeaways; they should act on every bug report, but can and will ignore bad reports
312. takeaways; act accordingly and sent a decent report, then you'll be heard [to do that, you]
313. takeaways; (1) check what kind of issue you deal with, as it…
314. takeaways; (a) might save you from wasting time on reporting known deficits
315. takeaways; (b) tells you what to expect from developers [in addition to that]
316. takeaways; (2) do your homework
317. takeaways; (a) test and report with a *vanilla* kernel
318. takeaways; (b) test with a fresh mainline kernel
319. takeaways; (c) rule out local interferences
320. takeaways; (d) check MAINTAINERS to submit the report to the right place
321. takeaways; (e) write a friendly and decent report easy to gasp for others
322. https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html * reporting issues takes care of this
323. takeaways; chances then are pretty good someone will help you
324. takeaways; and nearly perfect, if you report a bisected regression [and in the end that is]
325. takeaways; that's how you make the Linux developers fix kernel bugs they are able to fix [which is in everybody's interest and makes everyone happy]
327. questions?
328. Thorsten Leemhuis mail: linux@leemhuis.info GPG Key: 0x72B6E6EF4C583D2D social media: @kernellogger, @knurd42, @knurd42rhfc, @thleemhuis and @thleemhuisfoss on #twitter & #friendica #EOF