1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
3 If you want to distribute this text under CC-BY-4.0 only, please use 'The
4 Linux kernel developers' for author attribution and link this as source:
5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst
7 Note: Only the content of this RST file as found in the Linux kernel sources
8 is available under CC-BY-4.0, as versions of this text that were processed
9 (for example by the kernel's build system) might contain content taken from
10 files which use a more restrictive license.
17 The short guide (aka TL;DR)
18 ===========================
20 Are you facing a regression with vanilla kernels from the same stable or
21 longterm series? One still supported? Then search the `LKML
22 <https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list
23 <https://lore.kernel.org/stable/>`_ archives for matching reports to join. If
24 you don't find any, install `the latest release from that series
25 <https://kernel.org/>`_. If it still shows the issue, report it to the stable
26 mailing list (stable@vger.kernel.org) and CC the regressions list
27 (regressions@lists.linux.dev).
29 In all other cases try your best guess which kernel part might be causing the
30 issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers
31 expect to be told about problems, which most of the time will be by email with a
32 mailing list in CC. Check the destination's archives for matching reports;
33 search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you
34 don't find any to join, install `the latest mainline kernel
35 <https://kernel.org/>`_. If the issue is present there, send a report.
37 The issue was fixed there, but you would like to see it resolved in a still
38 supported stable or longterm series as well? Then install its latest release.
39 If it shows the problem, search for the change that fixed it in mainline and
40 check if backporting is in the works or was discarded; if it's neither, ask
41 those who handled the change for it.
43 **General remarks**: When installing and testing a kernel as outlined above,
44 ensure it's vanilla (IOW: not patched and not using add-on modules). Also make
45 sure it's built and running in a healthy environment and not already tainted
46 before the issue occurs.
48 If you are facing multiple issues with the Linux kernel at once, report each
49 separately. While writing your report, include all information relevant to the
50 issue, like the kernel and the distro used. In case of a regression, CC the
51 regressions mailing list (regressions@lists.linux.dev) to your report; also try
52 to include the commit-id of the change causing it, which a bisection can find.
54 Once the report is out, answer any questions that come up and help where you
55 can. That includes keeping the ball rolling by occasionally retesting with newer
56 releases and sending a status update afterwards.
58 Step-by-step guide how to report issues to the kernel maintainers
59 =================================================================
61 The above TL;DR outlines roughly how to report issues to the Linux kernel
62 developers. It might be all that's needed for people already familiar with
63 reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For
64 everyone else there is this section. It is more detailed and uses a
65 step-by-step approach. It still tries to be brief for readability and leaves
66 out a lot of details; those are described below the step-by-step guide in a
67 reference section, which explains each of the steps in more detail.
69 Note: this section covers a few more aspects than the TL;DR and does things in
70 a slightly different order. That's in your interest, to make sure you notice
71 early if an issue that looks like a Linux kernel problem is actually caused by
72 something else. These steps thus help to ensure the time you invest in this
73 process won't feel wasted in the end:
75 * Are you facing an issue with a Linux kernel a hardware or software vendor
76 provided? Then in almost all cases you are better off to stop reading this
77 document and reporting the issue to your vendor instead, unless you are
78 willing to install the latest Linux version yourself. Be aware the latter
79 will often be needed anyway to hunt down and fix issues.
81 * Perform a rough search for existing reports with your favorite internet
82 search engine; additionally, check the archives of the `Linux Kernel Mailing
83 List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports,
84 join the discussion instead of sending a new one.
86 * See if the issue you are dealing with qualifies as regression, security
87 issue, or a really severe problem: those are 'issues of high priority' that
88 need special handling in some steps that are about to follow.
90 * Make sure it's not the kernel's surroundings that are causing the issue
93 * Create a fresh backup and put system repair and restore tools at hand.
95 * Ensure your system does not enhance its kernels by building additional
96 kernel modules on-the-fly, which solutions like DKMS might be doing locally
97 without your knowledge.
99 * Check if your kernel was 'tainted' when the issue occurred, as the event
100 that made the kernel set this flag might be causing the issue you face.
102 * Write down coarsely how to reproduce the issue. If you deal with multiple
103 issues at once, create separate notes for each of them and make sure they
104 work independently on a freshly booted system. That's needed, as each issue
105 needs to get reported to the kernel developers separately, unless they are
108 * If you are facing a regression within a stable or longterm version line
109 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to
110 'Dealing with regressions within a stable and longterm kernel line'.
112 * Locate the driver or kernel subsystem that seems to be causing the issue.
113 Find out how and where its developers expect reports. Note: most of the
114 time this won't be bugzilla.kernel.org, as issues typically need to be sent
115 by mail to a maintainer and a public mailing list.
117 * Search the archives of the bug tracker or mailing list in question
118 thoroughly for reports that might match your issue. If you find anything,
119 join the discussion instead of sending a new report.
121 After these preparations you'll now enter the main part:
123 * Unless you are already running the latest 'mainline' Linux kernel, better
124 go and install it for the reporting process. Testing and reporting with
125 the latest 'stable' Linux can be an acceptable alternative in some
126 situations; during the merge window that actually might be even the best
127 approach, but in that development phase it can be an even better idea to
128 suspend your efforts for a few days anyway. Whatever version you choose,
129 ideally use a 'vanilla' build. Ignoring these advices will dramatically
130 increase the risk your report will be rejected or ignored.
132 * Ensure the kernel you just installed does not 'taint' itself when
135 * Reproduce the issue with the kernel you just installed. If it doesn't show
136 up there, scroll down to the instructions for issues only happening with
137 stable and longterm kernels.
139 * Optimize your notes: try to find and write the most straightforward way to
140 reproduce your issue. Make sure the end result has all the important
141 details, and at the same time is easy to read and understand for others
142 that hear about it for the first time. And if you learned something in this
143 process, consider searching again for existing reports about the issue.
145 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider
146 decoding the kernel log to find the line of code that triggered the error.
148 * If your problem is a regression, try to narrow down when the issue was
149 introduced as much as possible.
151 * Start to compile the report by writing a detailed description about the
152 issue. Always mention a few things: the latest kernel version you installed
153 for reproducing, the Linux Distribution used, and your notes on how to
154 reproduce the issue. Ideally, make the kernel's build configuration
155 (.config) and the output from ``dmesg`` available somewhere on the net and
156 link to it. Include or upload all other information that might be relevant,
157 like the output/screenshot of an Oops or the output from ``lspci``. Once
158 you wrote this main part, insert a normal length paragraph on top of it
159 outlining the issue and the impact quickly. On top of this add one sentence
160 that briefly describes the problem and gets people to read on. Now give the
161 thing a descriptive title or subject that yet again is shorter. Then you're
162 ready to send or file the report like the MAINTAINERS file told you, unless
163 you are dealing with one of those 'issues of high priority': they need
164 special care which is explained in 'Special handling for high priority
167 * Wait for reactions and keep the thing rolling until you can accept the
168 outcome in one way or the other. Thus react publicly and in a timely manner
169 to any inquiries. Test proposed fixes. Do proactive testing: retest with at
170 least every first release candidate (RC) of a new mainline version and
171 report your results. Send friendly reminders if things stall. And try to
172 help yourself, if you don't get any help or if it's unsatisfying.
175 Reporting regressions within a stable and longterm kernel line
176 --------------------------------------------------------------
178 This subsection is for you, if you followed above process and got sent here at
179 the point about regression within a stable or longterm kernel version line. You
180 face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a
181 switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such
182 regressions as quickly as possible, hence there is a streamlined process to
185 * Check if the kernel developers still maintain the Linux kernel version
186 line you care about: go to the `front page of kernel.org
187 <https://kernel.org/>`_ and make sure it mentions
188 the latest release of the particular version line without an '[EOL]' tag.
190 * Check the archives of the `Linux stable mailing list
191 <https://lore.kernel.org/stable/>`_ for existing reports.
193 * Install the latest release from the particular version line as a vanilla
194 kernel. Ensure this kernel is not tainted and still shows the problem, as
195 the issue might have already been fixed there. If you first noticed the
196 problem with a vendor kernel, check a vanilla build of the last version
197 known to work performs fine as well.
199 * Send a short problem report to the Linux stable mailing list
200 (stable@vger.kernel.org) and CC the Linux regressions mailing list
201 (regressions@lists.linux.dev). Roughly describe the issue and ideally
202 explain how to reproduce it. Mention the first version that shows the
203 problem and the last version that's working fine. Then wait for further
206 The reference section below explains each of these steps in more detail.
209 Reporting issues only occurring in older kernel version lines
210 -------------------------------------------------------------
212 This subsection is for you, if you tried the latest mainline kernel as outlined
213 above, but failed to reproduce your issue there; at the same time you want to
214 see the issue fixed in a still supported stable or longterm series or vendor
215 kernels regularly rebased on those. If that the case, follow these steps:
217 * Prepare yourself for the possibility that going through the next few steps
218 might not get the issue solved in older releases: the fix might be too big
219 or risky to get backported there.
221 * Perform the first three steps in the section "Dealing with regressions
222 within a stable and longterm kernel line" above.
224 * Search the Linux kernel version control system for the change that fixed
225 the issue in mainline, as its commit message might tell you if the fix is
226 scheduled for backporting already. If you don't find anything that way,
227 search the appropriate mailing lists for posts that discuss such an issue
228 or peer-review possible fixes; then check the discussions if the fix was
229 deemed unsuitable for backporting. If backporting was not considered at
230 all, join the newest discussion, asking if it's in the cards.
232 * One of the former steps should lead to a solution. If that doesn't work
233 out, ask the maintainers for the subsystem that seems to be causing the
234 issue for advice; CC the mailing list for the particular subsystem as well
235 as the stable mailing list.
237 The reference section below explains each of these steps in more detail.
240 Reference section: Reporting issues to the kernel maintainers
241 =============================================================
243 The detailed guides above outline all the major steps in brief fashion, which
244 should be enough for most people. But sometimes there are situations where even
245 experienced users might wonder how to actually do one of those steps. That's
246 what this section is for, as it will provide a lot more details on each of the
247 above steps. Consider this as reference documentation: it's possible to read it
248 from top to bottom. But it's mainly meant to skim over and a place to look up
249 details how to actually perform those steps.
251 A few words of general advice before digging into the details:
253 * The Linux kernel developers are well aware this process is complicated and
254 demands more than other FLOSS projects. We'd love to make it simpler. But
255 that would require work in various places as well as some infrastructure,
256 which would need constant maintenance; nobody has stepped up to do that
257 work, so that's just how things are for now.
259 * A warranty or support contract with some vendor doesn't entitle you to
260 request fixes from developers in the upstream Linux kernel community: such
261 contracts are completely outside the scope of the Linux kernel, its
262 development community, and this document. That's why you can't demand
263 anything such a contract guarantees in this context, not even if the
264 developer handling the issue works for the vendor in question. If you want
265 to claim your rights, use the vendor's support channel instead. When doing
266 so, you might want to mention you'd like to see the issue fixed in the
267 upstream Linux kernel; motivate them by saying it's the only way to ensure
268 the fix in the end will get incorporated in all Linux distributions.
270 * If you never reported an issue to a FLOSS project before you should consider
271 reading `How to Report Bugs Effectively
272 <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_, `How To Ask
273 Questions The Smart Way
274 <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to ask good
275 questions <https://jvns.ca/blog/good-questions/>`_.
277 With that off the table, find below the details on how to properly report
278 issues to the Linux kernel developers.
281 Make sure you're using the upstream Linux kernel
282 ------------------------------------------------
284 *Are you facing an issue with a Linux kernel a hardware or software vendor
285 provided? Then in almost all cases you are better off to stop reading this
286 document and reporting the issue to your vendor instead, unless you are
287 willing to install the latest Linux version yourself. Be aware the latter
288 will often be needed anyway to hunt down and fix issues.*
290 Like most programmers, Linux kernel developers don't like to spend time dealing
291 with reports for issues that don't even happen with their current code. It's
292 just a waste everybody's time, especially yours. Unfortunately such situations
293 easily happen when it comes to the kernel and often leads to frustration on both
294 sides. That's because almost all Linux-based kernels pre-installed on devices
295 (Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux
296 distributors are quite distant from the official Linux kernel as distributed by
297 kernel.org: these kernels from these vendors are often ancient from the point of
298 Linux development or heavily modified, often both.
300 Most of these vendor kernels are quite unsuitable for reporting issues to the
301 Linux kernel developers: an issue you face with one of them might have been
302 fixed by the Linux kernel developers months or years ago already; additionally,
303 the modifications and enhancements by the vendor might be causing the issue you
304 face, even if they look small or totally unrelated. That's why you should report
305 issues with these kernels to the vendor. Its developers should look into the
306 report and, in case it turns out to be an upstream issue, fix it directly
307 upstream or forward the report there. In practice that often does not work out
308 or might not what you want. You thus might want to consider circumventing the
309 vendor by installing the very latest Linux kernel core yourself. If that's an
310 option for you move ahead in this process, as a later step in this guide will
311 explain how to do that once it rules out other potential causes for your issue.
313 Note, the previous paragraph is starting with the word 'most', as sometimes
314 developers in fact are willing to handle reports about issues occurring with
315 vendor kernels. If they do in the end highly depends on the developers and the
316 issue in question. Your chances are quite good if the distributor applied only
317 small modifications to a kernel based on a recent Linux version; that for
318 example often holds true for the mainline kernels shipped by Debian GNU/Linux
319 Sid or Fedora Rawhide. Some developers will also accept reports about issues
320 with kernels from distributions shipping the latest stable kernel, as long as
321 its only slightly modified; that for example is often the case for Arch Linux,
322 regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better
323 want to use a mainline Linux and avoid using a stable kernel for this
324 process, as outlined in the section 'Install a fresh kernel for testing' in more
327 Obviously you are free to ignore all this advice and report problems with an old
328 or heavily modified vendor kernel to the upstream Linux developers. But note,
329 those often get rejected or ignored, so consider yourself warned. But it's still
330 better than not reporting the issue at all: sometimes such reports directly or
331 indirectly will help to get the issue fixed over time.
334 Search for existing reports, first run
335 --------------------------------------
337 *Perform a rough search for existing reports with your favorite internet
338 search engine; additionally, check the archives of the Linux Kernel Mailing
339 List (LKML). If you find matching reports, join the discussion instead of
342 Reporting an issue that someone else already brought forward is often a waste of
343 time for everyone involved, especially you as the reporter. So it's in your own
344 interest to thoroughly check if somebody reported the issue already. At this
345 step of the process it's okay to just perform a rough search: a later step will
346 tell you to perform a more detailed search once you know where your issue needs
347 to be reported to. Nevertheless, do not hurry with this step of the reporting
348 process, it can save you time and trouble.
350 Simply search the internet with your favorite search engine first. Afterwards,
351 search the `Linux Kernel Mailing List (LKML) archives
352 <https://lore.kernel.org/lkml/>`_.
354 If you get flooded with results consider telling your search engine to limit
355 search timeframe to the past month or year. And wherever you search, make sure
356 to use good search terms; vary them a few times, too. While doing so try to
357 look at the issue from the perspective of someone else: that will help you to
358 come up with other words to use as search terms. Also make sure not to use too
359 many search terms at once. Remember to search with and without information like
360 the name of the kernel driver or the name of the affected hardware component.
361 But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC')
362 often is not much helpful, as it is too specific. Instead try search terms like
363 the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip
364 ('Navi' or 'Navi10') with and without its manufacturer ('AMD').
366 In case you find an existing report about your issue, join the discussion, as
367 you might be able to provide valuable additional information. That can be
368 important even when a fix is prepared or in its final stages already, as
369 developers might look for people that can provide additional information or
370 test a proposed fix. Jump to the section 'Duties after the report went out' for
371 details on how to get properly involved.
373 Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also
374 be a good idea, as that might provide valuable insights or turn up matching
375 reports. If you find the latter, just keep in mind: most subsystems expect
376 reports in different places, as described below in the section "Check where you
377 need to report your issue". The developers that should take care of the issue
378 thus might not even be aware of the bugzilla ticket. Hence, check the ticket if
379 the issue already got reported as outlined in this document and if not consider
383 Issue of high priority?
384 -----------------------
386 *See if the issue you are dealing with qualifies as regression, security
387 issue, or a really severe problem: those are 'issues of high priority' that
388 need special handling in some steps that are about to follow.*
390 Linus Torvalds and the leading Linux kernel developers want to see some issues
391 fixed as soon as possible, hence there are 'issues of high priority' that get
392 handled slightly differently in the reporting process. Three type of cases
393 qualify: regressions, security issues, and really severe problems.
395 You deal with a 'regression' if something that worked with an older version of
396 the Linux kernel does not work with a newer one or somehow works worse with it.
397 It thus is a regression when a WiFi driver that did a fine job with Linux 5.7
398 somehow misbehaves with 5.8 or doesn't work at all. It's also a regression if
399 an application shows erratic behavior with a newer kernel, which might happen
400 due to incompatible changes in the interface between the kernel and the
401 userland (like procfs and sysfs). Significantly reduced performance or
402 increased power consumption also qualify as regression. But keep in mind: the
403 new kernel needs to be built with a configuration that is similar to the one
404 from the old kernel (see below how to achieve that). That's because the kernel
405 developers sometimes can not avoid incompatibilities when implementing new
406 features; but to avoid regressions such features have to be enabled explicitly
407 during build time configuration.
409 What qualifies as security issue is left to your judgment. Consider reading
410 'Documentation/admin-guide/security-bugs.rst' before proceeding, as it
411 provides additional details how to best handle security issues.
413 An issue is a 'really severe problem' when something totally unacceptably bad
414 happens. That's for example the case when a Linux kernel corrupts the data it's
415 handling or damages hardware it's running on. You're also dealing with a severe
416 issue when the kernel suddenly stops working with an error message ('kernel
417 panic') or without any farewell note at all. Note: do not confuse a 'panic' (a
418 fatal error where the kernel stop itself) with a 'Oops' (a recoverable error),
419 as the kernel remains running after the latter.
422 Ensure a healthy environment
423 ----------------------------
425 *Make sure it's not the kernel's surroundings that are causing the issue
428 Problems that look a lot like a kernel issue are sometimes caused by build or
429 runtime environment. It's hard to rule out that problem completely, but you
432 * Use proven tools when building your kernel, as bugs in the compiler or the
433 binutils can cause the resulting kernel to misbehave.
435 * Ensure your computer components run within their design specifications;
436 that's especially important for the main processor, the main memory, and the
437 motherboard. Therefore, stop undervolting or overclocking when facing a
438 potential kernel issue.
440 * Try to make sure it's not faulty hardware that is causing your issue. Bad
441 main memory for example can result in a multitude of issues that will
442 manifest itself in problems looking like kernel issues.
444 * If you're dealing with a filesystem issue, you might want to check the file
445 system in question with ``fsck``, as it might be damaged in a way that leads
446 to unexpected kernel behavior.
448 * When dealing with a regression, make sure it's not something else that
449 changed in parallel to updating the kernel. The problem for example might be
450 caused by other software that was updated at the same time. It can also
451 happen that a hardware component coincidentally just broke when you rebooted
452 into a new kernel for the first time. Updating the systems BIOS or changing
453 something in the BIOS Setup can also lead to problems that on look a lot
454 like a kernel regression.
457 Prepare for emergencies
458 -----------------------
460 *Create a fresh backup and put system repair and restore tools at hand.*
462 Reminder, you are dealing with computers, which sometimes do unexpected things,
463 especially if you fiddle with crucial parts like the kernel of its operating
464 system. That's what you are about to do in this process. Thus, make sure to
465 create a fresh backup; also ensure you have all tools at hand to repair or
466 reinstall the operating system as well as everything you need to restore the
470 Make sure your kernel doesn't get enhanced
471 ------------------------------------------
473 *Ensure your system does not enhance its kernels by building additional
474 kernel modules on-the-fly, which solutions like DKMS might be doing locally
475 without your knowledge.*
477 The risk your issue report gets ignored or rejected dramatically increases if
478 your kernel gets enhanced in any way. That's why you should remove or disable
479 mechanisms like akmods and DKMS: those build add-on kernel modules
480 automatically, for example when you install a new Linux kernel or boot it for
481 the first time. Also remove any modules they might have installed. Then reboot
484 Note, you might not be aware that your system is using one of these solutions:
485 they often get set up silently when you install Nvidia's proprietary graphics
486 driver, VirtualBox, or other software that requires a some support from a
487 module not part of the Linux kernel. That why your might need to uninstall the
488 packages with such software to get rid of any 3rd party kernel module.
494 *Check if your kernel was 'tainted' when the issue occurred, as the event
495 that made the kernel set this flag might be causing the issue you face.*
497 The kernel marks itself with a 'taint' flag when something happens that might
498 lead to follow-up errors that look totally unrelated. The issue you face might
499 be such an error if your kernel is tainted. That's why it's in your interest to
500 rule this out early before investing more time into this process. This is the
501 only reason why this step is here, as this process later will tell you to
502 install the latest mainline kernel; you will need to check the taint flag again
503 then, as that's when it matters because it's the kernel the report will focus
506 On a running system is easy to check if the kernel tainted itself: if ``cat
507 /proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and
508 everything is fine. Checking that file is impossible in some situations; that's
509 why the kernel also mentions the taint status when it reports an internal
510 problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a
511 non-recoverable error before halting operation (a 'kernel panic'). Look near
512 the top of the error messages printed when one of these occurs and search for a
513 line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was
514 not tainted when it noticed the problem; it was tainted if you see 'Tainted:'
515 followed by a few spaces and some letters.
517 If your kernel is tainted, study 'Documentation/admin-guide/tainted-kernels.rst'
518 to find out why. Try to eliminate the reason. Often it's caused by one these
521 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted
522 itself, as the kernel knows it might misbehave in strange ways after that
523 point. In that case check your kernel or system log and look for a section
524 that starts with this::
528 That's the first Oops since boot-up, as the '#1' between the brackets shows.
529 Every Oops and any other problem that happens after that point might be a
530 follow-up problem to that first Oops, even if both look totally unrelated.
531 Rule this out by getting rid of the cause for the first Oops and reproducing
532 the issue afterwards. Sometimes simply restarting will be enough, sometimes
533 a change to the configuration followed by a reboot can eliminate the Oops.
534 But don't invest too much time into this at this point of the process, as
535 the cause for the Oops might already be fixed in the newer Linux kernel
536 version you are going to install later in this process.
538 2. Your system uses a software that installs its own kernel modules, for
539 example Nvidia's proprietary graphics driver or VirtualBox. The kernel
540 taints itself when it loads such module from external sources (even if
541 they are Open Source): they sometimes cause errors in unrelated kernel
542 areas and thus might be causing the issue you face. You therefore have to
543 prevent those modules from loading when you want to report an issue to the
544 Linux kernel developers. Most of the time the easiest way to do that is:
545 temporarily uninstall such software including any modules they might have
546 installed. Afterwards reboot.
548 3. The kernel also taints itself when it's loading a module that resides in
549 the staging tree of the Linux kernel source. That's a special area for
550 code (mostly drivers) that does not yet fulfill the normal Linux kernel
551 quality standards. When you report an issue with such a module it's
552 obviously okay if the kernel is tainted; just make sure the module in
553 question is the only reason for the taint. If the issue happens in an
554 unrelated area reboot and temporarily block the module from being loaded
555 by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with
556 the name of the module in question).
559 Document how to reproduce issue
560 -------------------------------
562 *Write down coarsely how to reproduce the issue. If you deal with multiple
563 issues at once, create separate notes for each of them and make sure they
564 work independently on a freshly booted system. That's needed, as each issue
565 needs to get reported to the kernel developers separately, unless they are
568 If you deal with multiple issues at once, you'll have to report each of them
569 separately, as they might be handled by different developers. Describing
570 various issues in one report also makes it quite difficult for others to tear
571 it apart. Hence, only combine issues in one report if they are very strongly
574 Additionally, during the reporting process you will have to test if the issue
575 happens with other kernel versions. Therefore, it will make your work easier if
576 you know exactly how to reproduce an issue quickly on a freshly booted system.
578 Note: it's often fruitless to report issues that only happened once, as they
579 might be caused by a bit flip due to cosmic radiation. That's why you should
580 try to rule that out by reproducing the issue before going further. Feel free
581 to ignore this advice if you are experienced enough to tell a one-time error
582 due to faulty hardware apart from a kernel issue that rarely happens and thus
583 is hard to reproduce.
586 Regression in stable or longterm kernel?
587 ----------------------------------------
589 *If you are facing a regression within a stable or longterm version line
590 (say something broke when updating from 5.10.4 to 5.10.5), scroll down to
591 'Dealing with regressions within a stable and longterm kernel line'.*
593 Regression within a stable and longterm kernel version line are something the
594 Linux developers want to fix badly, as such issues are even more unwanted than
595 regression in the main development branch, as they can quickly affect a lot of
596 people. The developers thus want to learn about such issues as quickly as
597 possible, hence there is a streamlined process to report them. Note,
598 regressions with newer kernel version line (say something broke when switching
599 from 5.9.15 to 5.10.5) do not qualify.
602 Check where you need to report your issue
603 -----------------------------------------
605 *Locate the driver or kernel subsystem that seems to be causing the issue.
606 Find out how and where its developers expect reports. Note: most of the
607 time this won't be bugzilla.kernel.org, as issues typically need to be sent
608 by mail to a maintainer and a public mailing list.*
610 It's crucial to send your report to the right people, as the Linux kernel is a
611 big project and most of its developers are only familiar with a small subset of
612 it. Quite a few programmers for example only care for just one driver, for
613 example one for a WiFi chip; its developer likely will only have small or no
614 knowledge about the internals of remote or unrelated "subsystems", like the TCP
615 stack, the PCIe/PCI subsystem, memory management or file systems.
617 Problem is: the Linux kernel lacks a central bug tracker where you can simply
618 file your issue and make it reach the developers that need to know about it.
619 That's why you have to find the right place and way to report issues yourself.
620 You can do that with the help of a script (see below), but it mainly targets
621 kernel developers and experts. For everybody else the MAINTAINERS file is the
624 How to read the MAINTAINERS file
625 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
626 To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, lets assume
627 the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that
628 case it's likely an issue in the WiFi driver. Obviously it could also be some
629 code it builds upon, but unless you suspect something like that stick to the
630 driver. If it's really something else, the driver's developers will get the
631 right people involved.
633 Sadly, there is no way to check which code is driving a particular hardware
634 component that is both universal and easy.
636 In case of a problem with the WiFi driver you for example might want to look at
637 the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the
638 kernel module driving it::
640 [user@something ~]$ lspci -k
642 3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
643 Subsystem: Bigfoot Networks, Inc. Device 1535
644 Kernel driver in use: ath10k_pci
645 Kernel modules: ath10k_pci
648 But this approach won't work if your WiFi chip is connected over USB or some
649 other internal bus. In those cases you might want to check your WiFi manager or
650 the output of ``ip link``. Look for the name of the problematic network
651 interface, which might be something like 'wlp58s0'. This name can be used like
652 this to find the module driving it::
654 [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module
657 In case tricks like these don't bring you any further, try to search the
658 internet on how to narrow down the driver or subsystem in question. And if you
659 are unsure which it is: just try your best guess, somebody will help you if you
662 Once you know the driver or subsystem, you want to search for it in the
663 MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the
664 name is too specific. Sometimes you will need to search on the net for help;
665 but before doing so, try a somewhat shorted or modified name when searching the
666 MAINTAINERS file, as then you might find something like this::
668 QUALCOMM ATHEROS ATH10K WIRELESS DRIVER
669 Mail: A. Some Human <shuman@example.com>
670 Mailing list: ath10k@lists.infradead.org
672 Web-page: https://wireless.wiki.kernel.org/en/users/Drivers/ath10k
673 SCM: git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git
674 Files: drivers/net/wireless/ath/ath10k/
676 Note: the line description will be abbreviations, if you read the plain
677 MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for
678 example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'.
679 A section near the top of the file explains these and other abbreviations.
681 First look at the line 'Status'. Ideally it should be 'Supported' or
682 'Maintained'. If it states 'Obsolete' then you are using some outdated approach
683 that was replaced by a newer solution you need to switch to. Sometimes the code
684 only has someone who provides 'Odd Fixes' when feeling motivated. And with
685 'Orphan' you are totally out of luck, as nobody takes care of the code anymore.
686 That only leaves these options: arrange yourself to live with the issue, fix it
687 yourself, or find a programmer somewhere willing to fix it.
689 After checking the status, look for a line starting with 'bugs:': it will tell
690 you where to find a subsystem specific bug tracker to file your issue. The
691 example above does not have such a line. That is the case for most sections, as
692 Linux kernel development is completely driven by mail. Very few subsystems use
693 a bug tracker, and only some of those rely on bugzilla.kernel.org.
695 In this and many other cases you thus have to look for lines starting with
696 'Mail:' instead. Those mention the name and the email addresses for the
697 maintainers of the particular code. Also look for a line starting with 'Mailing
698 list:', which tells you the public mailing list where the code is developed.
699 Your report later needs to go by mail to those addresses. Additionally, for all
700 issue reports sent by email, make sure to add the Linux Kernel Mailing List
701 (LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing
702 lists when sending your issue report by mail later! Maintainers are busy people
703 and might leave some work for other developers on the subsystem specific list;
704 and LKML is important to have one place where all issue reports can be found.
707 Finding the maintainers with the help of a script
708 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
710 For people that have the Linux sources at hand there is a second option to find
711 the proper place to report: the script 'scripts/get_maintainer.pl' which tries
712 to find all people to contact. It queries the MAINTAINERS file and needs to be
713 called with a path to the source code in question. For drivers compiled as
714 module if often can be found with a command like this::
716 $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!'
717 drivers/net/wireless/ath/ath10k/ath10k_pci.ko
719 Pass parts of this to the script::
721 $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k*
722 Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
723 Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS)
724 ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
725 linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS))
726 netdev@vger.kernel.org (open list:NETWORKING DRIVERS)
727 linux-kernel@vger.kernel.org (open list)
729 Don't sent your report to all of them. Send it to the maintainers, which the
730 script calls "supporter:"; additionally CC the most specific mailing list for
731 the code as well as the Linux Kernel Mailing List (LKML). In this case you thus
732 would need to send the report to 'Some Human <shuman@example.com>' with
733 'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC.
735 Note: in case you cloned the Linux sources with git you might want to call
736 ``get_maintainer.pl`` a second time with ``--git``. The script then will look
737 at the commit history to find which people recently worked on the code in
738 question, as they might be able to help. But use these results with care, as it
739 can easily send you in a wrong direction. That for example happens quickly in
740 areas rarely changed (like old or unmaintained drivers): sometimes such code is
741 modified during tree-wide cleanups by developers that do not care about the
742 particular driver at all.
745 Search for existing reports, second run
746 ---------------------------------------
748 *Search the archives of the bug tracker or mailing list in question
749 thoroughly for reports that might match your issue. If you find anything,
750 join the discussion instead of sending a new report.*
752 As mentioned earlier already: reporting an issue that someone else already
753 brought forward is often a waste of time for everyone involved, especially you
754 as the reporter. That's why you should search for existing report again, now
755 that you know where they need to be reported to. If it's mailing list, you will
756 often find its archives on `lore.kernel.org <https://lore.kernel.org/>`_.
758 But some list are hosted in different places. That for example is the case for
759 the ath10k WiFi driver used as example in the previous step. But you'll often
760 find the archives for these lists easily on the net. Searching for 'archive
761 ath10k@lists.infradead.org' for example will lead you to the `Info page for the
762 ath10k mailing list <https://lists.infradead.org/mailman/listinfo/ath10k>`_,
763 which at the top links to its
764 `list archives <https://lists.infradead.org/pipermail/ath10k/>`_. Sadly this and
765 quite a few other lists miss a way to search the archives. In those cases use a
766 regular internet search engine and add something like
767 'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits
768 the results to the archives at that URL.
770 It's also wise to check the internet, LKML and maybe bugzilla.kernel.org again
773 For details how to search and what to do if you find matching reports see
774 "Search for existing reports, first run" above.
776 Do not hurry with this step of the reporting process: spending 30 to 60 minutes
777 or even more time can save you and others quite a lot of time and trouble.
780 Install a fresh kernel for testing
781 ----------------------------------
783 *Unless you are already running the latest 'mainline' Linux kernel, better
784 go and install it for the reporting process. Testing and reporting with
785 the latest 'stable' Linux can be an acceptable alternative in some
786 situations; during the merge window that actually might be even the best
787 approach, but in that development phase it can be an even better idea to
788 suspend your efforts for a few days anyway. Whatever version you choose,
789 ideally use a 'vanilla' built. Ignoring these advices will dramatically
790 increase the risk your report will be rejected or ignored.*
792 As mentioned in the detailed explanation for the first step already: Like most
793 programmers, Linux kernel developers don't like to spend time dealing with
794 reports for issues that don't even happen with the current code. It's just a
795 waste everybody's time, especially yours. That's why it's in everybody's
796 interest that you confirm the issue still exists with the latest upstream code
797 before reporting it. You are free to ignore this advice, but as outlined
798 earlier: doing so dramatically increases the risk that your issue report might
799 get rejected or simply ignored.
801 In the scope of the kernel "latest upstream" normally means:
803 * Install a mainline kernel; the latest stable kernel can be an option, but
804 most of the time is better avoided. Longterm kernels (sometimes called 'LTS
805 kernels') are unsuitable at this point of the process. The next subsection
806 explains all of this in more detail.
808 * The over next subsection describes way to obtain and install such a kernel.
809 It also outlines that using a pre-compiled kernel are fine, but better are
810 vanilla, which means: it was built using Linux sources taken straight `from
811 kernel.org <https://kernel.org/>`_ and not modified or enhanced in any way.
813 Choosing the right version for testing
814 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
816 Head over to `kernel.org <https://kernel.org/>`_ to find out which version you
817 want to use for testing. Ignore the big yellow button that says 'Latest release'
818 and look a little lower at the table. At its top you'll see a line starting with
819 mainline, which most of the time will point to a pre-release with a version
820 number like '5.8-rc2'. If that's the case, you'll want to use this mainline
821 kernel for testing, as that where all fixes have to be applied first. Do not let
822 that 'rc' scare you, these 'development kernels' are pretty reliable — and you
823 made a backup, as you were instructed above, didn't you?
825 In about two out of every nine to ten weeks, mainline might point you to a
826 proper release with a version number like '5.7'. If that happens, consider
827 suspending the reporting process until the first pre-release of the next
828 version (5.8-rc1) shows up on kernel.org. That's because the Linux development
829 cycle then is in its two-week long 'merge window'. The bulk of the changes and
830 all intrusive ones get merged for the next release during this time. It's a bit
831 more risky to use mainline during this period. Kernel developers are also often
832 quite busy then and might have no spare time to deal with issue reports. It's
833 also quite possible that one of the many changes applied during the merge
834 window fixes the issue you face; that's why you soon would have to retest with
835 a newer kernel version anyway, as outlined below in the section 'Duties after
836 the report went out'.
838 That's why it might make sense to wait till the merge window is over. But don't
839 to that if you're dealing with something that shouldn't wait. In that case
840 consider obtaining the latest mainline kernel via git (see below) or use the
841 latest stable version offered on kernel.org. Using that is also acceptable in
842 case mainline for some reason does currently not work for you. An in general:
843 using it for reproducing the issue is also better than not reporting it issue
846 Better avoid using the latest stable kernel outside merge windows, as all fixes
847 must be applied to mainline first. That's why checking the latest mainline
848 kernel is so important: any issue you want to see fixed in older version lines
849 needs to be fixed in mainline first before it can get backported, which can
850 take a few days or weeks. Another reason: the fix you hope for might be too
851 hard or risky for backporting; reporting the issue again hence is unlikely to
854 These aspects are also why longterm kernels (sometimes called "LTS kernels")
855 are unsuitable for this part of the reporting process: they are to distant from
856 the current code. Hence go and test mainline first and follow the process
857 further: if the issue doesn't occur with mainline it will guide you how to get
858 it fixed in older version lines, if that's in the cards for the fix in question.
860 How to obtain a fresh Linux kernel
861 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
863 **Using a pre-compiled kernel**: This is often the quickest, easiest, and safest
864 way for testing — especially is you are unfamiliar with the Linux kernel. The
865 problem: most of those shipped by distributors or add-on repositories are build
866 from modified Linux sources. They are thus not vanilla and therefore often
867 unsuitable for testing and issue reporting: the changes might cause the issue
868 you face or influence it somehow.
870 But you are in luck if you are using a popular Linux distribution: for quite a
871 few of them you'll find repositories on the net that contain packages with the
872 latest mainline or stable Linux built as vanilla kernel. It's totally okay to
873 use these, just make sure from the repository's description they are vanilla or
874 at least close to it. Additionally ensure the packages contain the latest
875 versions as offered on kernel.org. The packages are likely unsuitable if they
876 are older than a week, as new mainline and stable kernels typically get released
877 at least once a week.
879 Please note that you might need to build your own kernel manually later: that's
880 sometimes needed for debugging or testing fixes, as described later in this
881 document. Also be aware that pre-compiled kernels might lack debug symbols that
882 are needed to decode messages the kernel prints when a panic, Oops, warning, or
883 BUG occurs; if you plan to decode those, you might be better off compiling a
884 kernel yourself (see the end of this subsection and the section titled 'Decode
885 failure messages' for details).
887 **Using git**: Developers and experienced Linux users familiar with git are
888 often best served by obtaining the latest Linux kernel sources straight from the
889 `official development repository on kernel.org
890 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_.
891 Those are likely a bit ahead of the latest mainline pre-release. Don't worry
892 about it: they are as reliable as a proper pre-release, unless the kernel's
893 development cycle is currently in the middle of a merge window. But even then
894 they are quite reliable.
896 **Conventional**: People unfamiliar with git are often best served by
897 downloading the sources as tarball from `kernel.org <https://kernel.org/>`_.
899 How to actually build a kernel is not described here, as many websites explain
900 the necessary steps already. If you are new to it, consider following one of
901 those how-to's that suggest to use ``make localmodconfig``, as that tries to
902 pick up the configuration of your current kernel and then tries to adjust it
903 somewhat for your system. That does not make the resulting kernel any better,
904 but quicker to compile.
906 Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel,
907 please try to enable CONFIG_KALLSYMS when configuring your kernel.
908 Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the
909 latter is the relevant one of those two, but can only be reached if you enable
910 the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to
911 build a kernel by quite a bit. But that's worth it, as these options will allow
912 you later to pinpoint the exact line of code that triggers your issue. The
913 section 'Decode failure messages' below explains this in more detail.
915 But keep in mind: Always keep a record of the issue encountered in case it is
916 hard to reproduce. Sending an undecoded report is better than not reporting
923 *Ensure the kernel you just installed does not 'taint' itself when
926 As outlined above in more detail already: the kernel sets a 'taint' flag when
927 something happens that can lead to follow-up errors that look totally
928 unrelated. That's why you need to check if the kernel you just installed does
929 not set this flag. And if it does, you in almost all the cases needs to
930 eliminate the reason for it before you reporting issues that occur with it. See
931 the section above for details how to do that.
934 Reproduce issue with the fresh kernel
935 -------------------------------------
937 *Reproduce the issue with the kernel you just installed. If it doesn't show
938 up there, scroll down to the instructions for issues only happening with
939 stable and longterm kernels.*
941 Check if the issue occurs with the fresh Linux kernel version you just
942 installed. If it was fixed there already, consider sticking with this version
943 line and abandoning your plan to report the issue. But keep in mind that other
944 users might still be plagued by it, as long as it's not fixed in either stable
945 and longterm version from kernel.org (and thus vendor kernels derived from
946 those). If you prefer to use one of those or just want to help their users,
947 head over to the section "Details about reporting issues only occurring in
948 older kernel version lines" below.
951 Optimize description to reproduce issue
952 ---------------------------------------
954 *Optimize your notes: try to find and write the most straightforward way to
955 reproduce your issue. Make sure the end result has all the important
956 details, and at the same time is easy to read and understand for others
957 that hear about it for the first time. And if you learned something in this
958 process, consider searching again for existing reports about the issue.*
960 An unnecessarily complex report will make it hard for others to understand your
961 report. Thus try to find a reproducer that's straight forward to describe and
962 thus easy to understand in written form. Include all important details, but at
963 the same time try to keep it as short as possible.
965 In this in the previous steps you likely have learned a thing or two about the
966 issue you face. Use this knowledge and search again for existing reports
967 instead you can join.
970 Decode failure messages
971 -----------------------
973 *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider
974 decoding the kernel log to find the line of code that triggered the error.*
976 When the kernel detects an internal problem, it will log some information about
977 the executed code. This makes it possible to pinpoint the exact line in the
978 source code that triggered the issue and shows how it was called. But that only
979 works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring
980 your kernel. If you did so, consider to decode the information from the
981 kernel's log. That will make it a lot easier to understand what lead to the
982 'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone
985 Decoding can be done with a script you find in the Linux source tree. If you
986 are running a kernel you compiled yourself earlier, call it like this::
988 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux
990 If you are running a packaged vanilla kernel, you will likely have to install
991 the corresponding packages with debug symbols. Then call the script (which you
992 might need to get from the Linux sources if your distro does not package it)
995 [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \
996 /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/
998 The script will work on log lines like the following, which show the address of
999 the code the kernel was executing when the error occurred::
1001 [ 68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module]
1003 Once decoded, these lines will look like this::
1005 [ 68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module
1007 In this case the executed code was built from the file
1008 '~/linux-5.10.5/test-module/test-module.c' and the error occurred by the
1009 instructions found in line '16'.
1011 The script will similarly decode the addresses mentioned in the section
1012 starting with 'Call trace', which show the path to the function where the
1013 problem occurred. Additionally, the script will show the assembler output for
1014 the code section the kernel was executing.
1016 Note, if you can't get this to work, simply skip this step and mention the
1017 reason for it in the report. If you're lucky, it might not be needed. And if it
1018 is, someone might help you to get things going. Also be aware this is just one
1019 of several ways to decode kernel stack traces. Sometimes different steps will
1020 be required to retrieve the relevant details. Don't worry about that, if that's
1021 needed in your case, developers will tell you what to do.
1024 Special care for regressions
1025 ----------------------------
1027 *If your problem is a regression, try to narrow down when the issue was
1028 introduced as much as possible.*
1030 Linux lead developer Linus Torvalds insists that the Linux kernel never
1031 worsens, that's why he deems regressions as unacceptable and wants to see them
1032 fixed quickly. That's why changes that introduced a regression are often
1033 promptly reverted if the issue they cause can't get solved quickly any other
1034 way. Reporting a regression is thus a bit like playing a kind of trump card to
1035 get something quickly fixed. But for that to happen the change that's causing
1036 the regression needs to be known. Normally it's up to the reporter to track
1037 down the culprit, as maintainers often won't have the time or setup at hand to
1038 reproduce it themselves.
1040 To find the change there is a process called 'bisection' which the document
1041 'Documentation/admin-guide/bug-bisect.rst' describes in detail. That process
1042 will often require you to build about ten to twenty kernel images, trying to
1043 reproduce the issue with each of them before building the next. Yes, that takes
1044 some time, but don't worry, it works a lot quicker than most people assume.
1045 Thanks to a 'binary search' this will lead you to the one commit in the source
1046 code management system that's causing the regression. Once you find it, search
1047 the net for the subject of the change, its commit id and the shortened commit id
1048 (the first 12 characters of the commit id). This will lead you to existing
1049 reports about it, if there are any.
1051 Note, a bisection needs a bit of know-how, which not everyone has, and quite a
1052 bit of effort, which not everyone is willing to invest. Nevertheless, it's
1053 highly recommended performing a bisection yourself. If you really can't or
1054 don't want to go down that route at least find out which mainline kernel
1055 introduced the regression. If something for example breaks when switching from
1056 5.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6,
1057 5.7 and 5.8) to check when it first showed up. Unless you're trying to find a
1058 regression in a stable or longterm kernel, avoid testing versions which number
1059 has three sections (5.6.12, 5.7.8), as that makes the outcome hard to
1060 interpret, which might render your testing useless. Once you found the major
1061 version which introduced the regression, feel free to move on in the reporting
1062 process. But keep in mind: it depends on the issue at hand if the developers
1063 will be able to help without knowing the culprit. Sometimes they might
1064 recognize from the report want went wrong and can fix it; other times they will
1065 be unable to help unless you perform a bisection.
1067 When dealing with regressions make sure the issue you face is really caused by
1068 the kernel and not by something else, as outlined above already.
1070 In the whole process keep in mind: an issue only qualifies as regression if the
1071 older and the newer kernel got built with a similar configuration. The best way
1072 to archive this: copy the configuration file (``.config``) from the old working
1073 kernel freshly to each newer kernel version you try. Afterwards run ``make
1074 olddefconfig`` to adjust it for the needs of the new version.
1077 Write and send the report
1078 -------------------------
1080 *Start to compile the report by writing a detailed description about the
1081 issue. Always mention a few things: the latest kernel version you installed
1082 for reproducing, the Linux Distribution used, and your notes on how to
1083 reproduce the issue. Ideally, make the kernel's build configuration
1084 (.config) and the output from ``dmesg`` available somewhere on the net and
1085 link to it. Include or upload all other information that might be relevant,
1086 like the output/screenshot of an Oops or the output from ``lspci``. Once
1087 you wrote this main part, insert a normal length paragraph on top of it
1088 outlining the issue and the impact quickly. On top of this add one sentence
1089 that briefly describes the problem and gets people to read on. Now give the
1090 thing a descriptive title or subject that yet again is shorter. Then you're
1091 ready to send or file the report like the MAINTAINERS file told you, unless
1092 you are dealing with one of those 'issues of high priority': they need
1093 special care which is explained in 'Special handling for high priority
1096 Now that you have prepared everything it's time to write your report. How to do
1097 that is partly explained by the three documents linked to in the preface above.
1098 That's why this text will only mention a few of the essentials as well as
1099 things specific to the Linux kernel.
1101 There is one thing that fits both categories: the most crucial parts of your
1102 report are the title/subject, the first sentence, and the first paragraph.
1103 Developers often get quite a lot of mail. They thus often just take a few
1104 seconds to skim a mail before deciding to move on or look closer. Thus: the
1105 better the top section of your report, the higher are the chances that someone
1106 will look into it and help you. And that is why you should ignore them for now
1107 and write the detailed report first. ;-)
1109 Things each report should mention
1110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112 Describe in detail how your issue happens with the fresh vanilla kernel you
1113 installed. Try to include the step-by-step instructions you wrote and optimized
1114 earlier that outline how you and ideally others can reproduce the issue; in
1115 those rare cases where that's impossible try to describe what you did to
1118 Also include all the relevant information others might need to understand the
1119 issue and its environment. What's actually needed depends a lot on the issue,
1120 but there are some things you should include always:
1122 * the output from ``cat /proc/version``, which contains the Linux kernel
1123 version number and the compiler it was built with.
1125 * the Linux distribution the machine is running (``hostnamectl | grep
1126 "Operating System"``)
1128 * the architecture of the CPU and the operating system (``uname -mi``)
1130 * if you are dealing with a regression and performed a bisection, mention the
1131 subject and the commit-id of the change that is causing it.
1133 In a lot of cases it's also wise to make two more things available to those
1134 that read your report:
1136 * the configuration used for building your Linux kernel (the '.config' file)
1138 * the kernel's messages that you get from ``dmesg`` written to a file. Make
1139 sure that it starts with a line like 'Linux version 5.8-1
1140 (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug
1141 3 14:54:37 UTC 2020' If it's missing, then important messages from the first
1142 boot phase already got discarded. In this case instead consider using
1143 ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the
1144 issue and call ``dmesg`` right afterwards.
1146 These two files are big, that's why it's a bad idea to put them directly into
1147 your report. If you are filing the issue in a bug tracker then attach them to
1148 the ticket. If you report the issue by mail do not attach them, as that makes
1149 the mail too large; instead do one of these things:
1151 * Upload the files somewhere public (your website, a public file paste
1152 service, a ticket created just for this purpose on `bugzilla.kernel.org
1153 <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your
1154 report. Ideally use something where the files stay available for years, as
1155 they could be useful to someone many years from now; this for example can
1156 happen if five or ten years from now a developer works on some code that was
1157 changed just to fix your issue.
1159 * Put the files aside and mention you will send them later in individual
1160 replies to your own mail. Just remember to actually do that once the report
1163 Things that might be wise to provide
1164 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1166 Depending on the issue you might need to add more background data. Here are a
1167 few suggestions what often is good to provide:
1169 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel,
1170 include it. If you can't copy'n'paste it, try to capture a netconsole trace
1171 or at least take a picture of the screen.
1173 * If the issue might be related to your computer hardware, mention what kind
1174 of system you use. If you for example have problems with your graphics card,
1175 mention its manufacturer, the card's model, and what chip is uses. If it's a
1176 laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13'
1177 for example is not, because it might be the one from 2012; that one looks
1178 not that different from the one sold today, but apart from that the two have
1179 nothing in common. Hence, in such cases add the exact model number, which
1180 for example are '9380' or '7390' for XPS 13 models introduced during 2019.
1181 Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are
1182 variants of this laptop with and without a dedicated graphics chip, so try
1183 to find the exact model name or specify the main components.
1185 * Mention the relevant software in use. If you have problems with loading
1186 modules, you want to mention the versions of kmod, systemd, and udev in use.
1187 If one of the DRM drivers misbehaves, you want to state the versions of
1188 libdrm and Mesa; also specify your Wayland compositor or the X-Server and
1189 its driver. If you have a filesystem issue, mention the version of
1190 corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...).
1192 * Gather additional information from the kernel that might be of interest. The
1193 output from ``lspci -nn`` will for example help others to identify what
1194 hardware you use. If you have a problem with hardware you even might want to
1195 make the output from ``sudo lspci -vvv`` available, as that provides
1196 insights how the components were configured. For some issues it might be
1197 good to include the contents of files like ``/proc/cpuinfo``,
1198 ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or
1199 ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant
1200 information. One such tool is ``alsa-info.sh`` `which the audio/sound
1201 subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_.
1203 Those examples should give your some ideas of what data might be wise to
1204 attach, but you have to think yourself what will be helpful for others to know.
1205 Don't worry too much about forgetting something, as developers will ask for
1206 additional details they need. But making everything important available from
1207 the start increases the chance someone will take a closer look.
1210 The important part: the head of your report
1211 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1213 Now that you have the detailed part of the report prepared let's get to the
1214 most important section: the first few sentences. Thus go to the top, add
1215 something like 'The detailed description:' before the part you just wrote and
1216 insert two newlines at the top. Now write one normal length paragraph that
1217 describes the issue roughly. Leave out all boring details and focus on the
1218 crucial parts readers need to know to understand what this is all about; if you
1219 think this bug affects a lot of users, mention this to get people interested.
1221 Once you did that insert two more lines at the top and write a one sentence
1222 summary that explains quickly what the report is about. After that you have to
1223 get even more abstract and write an even shorter subject/title for the report.
1225 Now that you have written this part take some time to optimize it, as it is the
1226 most important parts of your report: a lot of people will only read this before
1227 they decide if reading the rest is time well spent.
1229 Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told
1230 you, unless it's one of those 'issues of high priority' outlined earlier: in
1231 that case please read the next subsection first before sending the report on
1234 Special handling for high priority issues
1235 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1237 Reports for high priority issues need special handling.
1239 **Severe issues**: make sure the subject or ticket title as well as the first
1240 paragraph makes the severeness obvious.
1242 **Regressions**: make the report's subject start with '[REGRESSION]'.
1244 In case you performed a successful bisection, use the title of the change that
1245 introduced the regression as the second part of your subject. Make the report
1246 also mention the commit id of the culprit. In case of an unsuccessful bisection,
1247 make your report mention the latest tested version that's working fine (say 5.7)
1248 and the oldest where the issue occurs (say 5.8-rc1).
1250 When sending the report by mail, CC the Linux regressions mailing list
1251 (regressions@lists.linux.dev). In case the report needs to be filed to some web
1252 tracker, proceed to do so; once filed, forward the report by mail to the
1253 regressions list. Make sure to inline the forwarded report, hence do not attach
1254 it. Also add a short note at the top where you mention the URL to the ticket.
1256 When mailing or forwarding the report, in case of a successful bisection add the
1257 author of the culprit to the recipients; also CC everyone in the signed-off-by
1258 chain, which you find at the end of its commit message.
1260 **Security issues**: for these issues your will have to evaluate if a
1261 short-term risk to other users would arise if details were publicly disclosed.
1262 If that's not the case simply proceed with reporting the issue as described.
1263 For issues that bear such a risk you will need to adjust the reporting process
1266 * If the MAINTAINERS file instructed you to report the issue by mail, do not
1267 CC any public mailing lists.
1269 * If you were supposed to file the issue in a bug tracker make sure to mark
1270 the ticket as 'private' or 'security issue'. If the bug tracker does not
1271 offer a way to keep reports private, forget about it and send your report as
1272 a private mail to the maintainers instead.
1274 In both cases make sure to also mail your report to the addresses the
1275 MAINTAINERS file lists in the section 'security contact'. Ideally directly CC
1276 them when sending the report by mail. If you filed it in a bug tracker, forward
1277 the report's text to these addresses; but on top of it put a small note where
1278 you mention that you filed it with a link to the ticket.
1280 See 'Documentation/admin-guide/security-bugs.rst' for more information.
1283 Duties after the report went out
1284 --------------------------------
1286 *Wait for reactions and keep the thing rolling until you can accept the
1287 outcome in one way or the other. Thus react publicly and in a timely manner
1288 to any inquiries. Test proposed fixes. Do proactive testing: retest with at
1289 least every first release candidate (RC) of a new mainline version and
1290 report your results. Send friendly reminders if things stall. And try to
1291 help yourself, if you don't get any help or if it's unsatisfying.*
1293 If your report was good and you are really lucky then one of the developers
1294 might immediately spot what's causing the issue; they then might write a patch
1295 to fix it, test it, and send it straight for integration in mainline while
1296 tagging it for later backport to stable and longterm kernels that need it. Then
1297 all you need to do is reply with a 'Thank you very much' and switch to a version
1298 with the fix once it gets released.
1300 But this ideal scenario rarely happens. That's why the job is only starting
1301 once you got the report out. What you'll have to do depends on the situations,
1302 but often it will be the things listed below. But before digging into the
1303 details, here are a few important things you need to keep in mind for this part
1307 General advice for further interactions
1308 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1310 **Always reply in public**: When you filed the issue in a bug tracker, always
1311 reply there and do not contact any of the developers privately about it. For
1312 mailed reports always use the 'Reply-all' function when replying to any mails
1313 you receive. That includes mails with any additional data you might want to add
1314 to your report: go to your mail applications 'Sent' folder and use 'reply-all'
1315 on your mail with the report. This approach will make sure the public mailing
1316 list(s) and everyone else that gets involved over time stays in the loop; it
1317 also keeps the mail thread intact, which among others is really important for
1318 mailing lists to group all related mails together.
1320 There are just two situations where a comment in a bug tracker or a 'Reply-all'
1323 * Someone tells you to send something privately.
1325 * You were told to send something, but noticed it contains sensitive
1326 information that needs to be kept private. In that case it's okay to send it
1327 in private to the developer that asked for it. But note in the ticket or a
1328 mail that you did that, so everyone else knows you honored the request.
1330 **Do research before asking for clarifications or help**: In this part of the
1331 process someone might tell you to do something that requires a skill you might
1332 not have mastered yet. For example, you might be asked to use some test tools
1333 you never have heard of yet; or you might be asked to apply a patch to the
1334 Linux kernel sources to test if it helps. In some cases it will be fine sending
1335 a reply asking for instructions how to do that. But before going that route try
1336 to find the answer own your own by searching the internet; alternatively
1337 consider asking in other places for advice. For example ask a friend or post
1338 about it to a chatroom or forum you normally hang out.
1340 **Be patient**: If you are really lucky you might get a reply to your report
1341 within a few hours. But most of the time it will take longer, as maintainers
1342 are scattered around the globe and thus might be in a different time zone – one
1343 where they already enjoy their night away from keyboard.
1345 In general, kernel developers will take one to five business days to respond to
1346 reports. Sometimes it will take longer, as they might be busy with the merge
1347 windows, other work, visiting developer conferences, or simply enjoying a long
1350 The 'issues of high priority' (see above for an explanation) are an exception
1351 here: maintainers should address them as soon as possible; that's why you
1352 should wait a week at maximum (or just two days if it's something urgent)
1353 before sending a friendly reminder.
1355 Sometimes the maintainer might not be responding in a timely manner; other
1356 times there might be disagreements, for example if an issue qualifies as
1357 regression or not. In such cases raise your concerns on the mailing list and
1358 ask others for public or private replies how to move on. If that fails, it
1359 might be appropriate to get a higher authority involved. In case of a WiFi
1360 driver that would be the wireless maintainers; if there are no higher level
1361 maintainers or all else fails, it might be one of those rare situations where
1362 it's okay to get Linus Torvalds involved.
1364 **Proactive testing**: Every time the first pre-release (the 'rc1') of a new
1365 mainline kernel version gets released, go and check if the issue is fixed there
1366 or if anything of importance changed. Mention the outcome in the ticket or in a
1367 mail you sent as reply to your report (make sure it has all those in the CC
1368 that up to that point participated in the discussion). This will show your
1369 commitment and that you are willing to help. It also tells developers if the
1370 issue persists and makes sure they do not forget about it. A few other
1371 occasional retests (for example with rc3, rc5 and the final) are also a good
1372 idea, but only report your results if something relevant changed or if you are
1373 writing something anyway.
1375 With all these general things off the table let's get into the details of how
1376 to help to get issues resolved once they were reported.
1378 Inquires and testing request
1379 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1381 Here are your duties in case you got replies to your report:
1383 **Check who you deal with**: Most of the time it will be the maintainer or a
1384 developer of the particular code area that will respond to your report. But as
1385 issues are normally reported in public it could be anyone that's replying —
1386 including people that want to help, but in the end might guide you totally off
1387 track with their questions or requests. That rarely happens, but it's one of
1388 many reasons why it's wise to quickly run an internet search to see who you're
1389 interacting with. By doing this you also get aware if your report was heard by
1390 the right people, as a reminder to the maintainer (see below) might be in order
1391 later if discussion fades out without leading to a satisfying solution for the
1394 **Inquiries for data**: Often you will be asked to test something or provide
1395 additional details. Try to provide the requested information soon, as you have
1396 the attention of someone that might help and risk losing it the longer you
1397 wait; that outcome is even likely if you do not provide the information within
1398 a few business days.
1400 **Requests for testing**: When you are asked to test a diagnostic patch or a
1401 possible fix, try to test it in timely manner, too. But do it properly and make
1402 sure to not rush it: mixing things up can happen easily and can lead to a lot
1403 of confusion for everyone involved. A common mistake for example is thinking a
1404 proposed patch with a fix was applied, but in fact wasn't. Things like that
1405 happen even to experienced testers occasionally, but they most of the time will
1406 notice when the kernel with the fix behaves just as one without it.
1408 What to do when nothing of substance happens
1409 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1411 Some reports will not get any reaction from the responsible Linux kernel
1412 developers; or a discussion around the issue evolved, but faded out with
1413 nothing of substance coming out of it.
1415 In these cases wait two (better: three) weeks before sending a friendly
1416 reminder: maybe the maintainer was just away from keyboard for a while when
1417 your report arrived or had something more important to take care of. When
1418 writing the reminder, kindly ask if anything else from your side is needed to
1419 get the ball running somehow. If the report got out by mail, do that in the
1420 first lines of a mail that is a reply to your initial mail (see above) which
1421 includes a full quote of the original report below: that's on of those few
1422 situations where such a 'TOFU' (Text Over, Fullquote Under) is the right
1423 approach, as then all the recipients will have the details at hand immediately
1424 in the proper order.
1426 After the reminder wait three more weeks for replies. If you still don't get a
1427 proper reaction, you first should reconsider your approach. Did you maybe try
1428 to reach out to the wrong people? Was the report maybe offensive or so
1429 confusing that people decided to completely stay away from it? The best way to
1430 rule out such factors: show the report to one or two people familiar with FLOSS
1431 issue reporting and ask for their opinion. Also ask them for their advice how
1432 to move forward. That might mean: prepare a better report and make those people
1433 review it before you send it out. Such an approach is totally fine; just
1434 mention that this is the second and improved report on the issue and include a
1435 link to the first report.
1437 If the report was proper you can send a second reminder; in it ask for advice
1438 why the report did not get any replies. A good moment for this second reminder
1439 mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel
1440 version got published, as you should retest and provide a status update at that
1441 point anyway (see above).
1443 If the second reminder again results in no reaction within a week, try to
1444 contact a higher-level maintainer asking for advice: even busy maintainers by
1445 then should at least have sent some kind of acknowledgment.
1447 Remember to prepare yourself for a disappointment: maintainers ideally should
1448 react somehow to every issue report, but they are only obliged to fix those
1449 'issues of high priority' outlined earlier. So don't be too devastating if you
1450 get a reply along the lines of 'thanks for the report, I have more important
1451 issues to deal with currently and won't have time to look into this for the
1452 foreseeable future'.
1454 It's also possible that after some discussion in the bug tracker or on a list
1455 nothing happens anymore and reminders don't help to motivate anyone to work out
1456 a fix. Such situations can be devastating, but is within the cards when it
1457 comes to Linux kernel development. This and several other reasons for not
1458 getting help are explained in 'Why some issues won't get any reaction or remain
1459 unfixed after being reported' near the end of this document.
1461 Don't get devastated if you don't find any help or if the issue in the end does
1462 not get solved: the Linux kernel is FLOSS and thus you can still help yourself.
1463 You for example could try to find others that are affected and team up with
1464 them to get the issue resolved. Such a team could prepare a fresh report
1465 together that mentions how many you are and why this is something that in your
1466 option should get fixed. Maybe together you can also narrow down the root cause
1467 or the change that introduced a regression, which often makes developing a fix
1468 easier. And with a bit of luck there might be someone in the team that knows a
1469 bit about programming and might be able to write a fix.
1472 Reference for "Reporting regressions within a stable and longterm kernel line"
1473 ------------------------------------------------------------------------------
1475 This subsection provides details for the steps you need to perform if you face
1476 a regression within a stable and longterm kernel line.
1478 Make sure the particular version line still gets support
1479 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1481 *Check if the kernel developers still maintain the Linux kernel version
1482 line you care about: go to the front page of kernel.org and make sure it
1483 mentions the latest release of the particular version line without an
1486 Most kernel version lines only get supported for about three months, as
1487 maintaining them longer is quite a lot of work. Hence, only one per year is
1488 chosen and gets supported for at least two years (often six). That's why you
1489 need to check if the kernel developers still support the version line you care
1492 Note, if kernel.org lists two stable version lines on the front page, you
1493 should consider switching to the newer one and forget about the older one:
1494 support for it is likely to be abandoned soon. Then it will get a "end-of-life"
1495 (EOL) stamp. Version lines that reached that point still get mentioned on the
1496 kernel.org front page for a week or two, but are unsuitable for testing and
1499 Search stable mailing list
1500 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1502 *Check the archives of the Linux stable mailing list for existing reports.*
1504 Maybe the issue you face is already known and was fixed or is about to. Hence,
1505 `search the archives of the Linux stable mailing list
1506 <https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If
1507 you find any matches, consider joining the discussion, unless the fix is
1508 already finished and scheduled to get applied soon.
1510 Reproduce issue with the newest release
1511 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1513 *Install the latest release from the particular version line as a vanilla
1514 kernel. Ensure this kernel is not tainted and still shows the problem, as
1515 the issue might have already been fixed there. If you first noticed the
1516 problem with a vendor kernel, check a vanilla build of the last version
1517 known to work performs fine as well.*
1519 Before investing any more time in this process you want to check if the issue
1520 was already fixed in the latest release of version line you're interested in.
1521 This kernel needs to be vanilla and shouldn't be tainted before the issue
1522 happens, as detailed outlined already above in the section "Install a fresh
1523 kernel for testing".
1525 Did you first notice the regression with a vendor kernel? Then changes the
1526 vendor applied might be interfering. You need to rule that out by performing
1527 a recheck. Say something broke when you updated from 5.10.4-vendor.42 to
1528 5.10.5-vendor.43. Then after testing the latest 5.10 release as outlined in
1529 the previous paragraph check if a vanilla build of Linux 5.10.4 works fine as
1530 well. If things are broken there, the issue does not qualify as upstream
1531 regression and you need switch back to the main step-by-step guide to report
1534 Report the regression
1535 ~~~~~~~~~~~~~~~~~~~~~
1537 *Send a short problem report to the Linux stable mailing list
1538 (stable@vger.kernel.org) and CC the Linux regressions mailing list
1539 (regressions@lists.linux.dev). Roughly describe the issue and ideally
1540 explain how to reproduce it. Mention the first version that shows the
1541 problem and the last version that's working fine. Then wait for further
1544 When reporting a regression that happens within a stable or longterm kernel
1545 line (say when updating from 5.10.4 to 5.10.5) a brief report is enough for
1546 the start to get the issue reported quickly. Hence a rough description is all
1549 But note, it helps developers a great deal if you can specify the exact version
1550 that introduced the problem. Hence if possible within a reasonable time frame,
1551 try to find that version using vanilla kernels. Lets assume something broke when
1552 your distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then as
1553 instructed above go and check the latest kernel from that version line, say
1554 5.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patches
1555 the distributor applied interfere. If the issue doesn't manifest itself there,
1556 try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find the
1557 first version where things broke. Mention it in the report and state that 5.10.9
1560 What the previous paragraph outlines is basically a rough manual 'bisection'.
1561 Once your report is out your might get asked to do a proper one, as it allows to
1562 pinpoint the exact change that causes the issue (which then can easily get
1563 reverted to fix the issue quickly). Hence consider to do a proper bisection
1564 right away if time permits. See the section 'Special care for regressions' and
1565 the document 'Documentation/admin-guide/bug-bisect.rst' for details how to
1569 Reference for "Reporting issues only occurring in older kernel version lines"
1570 -----------------------------------------------------------------------------
1572 This section provides details for the steps you need to take if you could not
1573 reproduce your issue with a mainline kernel, but want to see it fixed in older
1574 version lines (aka stable and longterm kernels).
1576 Some fixes are too complex
1577 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1579 *Prepare yourself for the possibility that going through the next few steps
1580 might not get the issue solved in older releases: the fix might be too big
1581 or risky to get backported there.*
1583 Even small and seemingly obvious code-changes sometimes introduce new and
1584 totally unexpected problems. The maintainers of the stable and longterm kernels
1585 are very aware of that and thus only apply changes to these kernels that are
1586 within rules outlined in 'Documentation/process/stable-kernel-rules.rst'.
1588 Complex or risky changes for example do not qualify and thus only get applied
1589 to mainline. Other fixes are easy to get backported to the newest stable and
1590 longterm kernels, but too risky to integrate into older ones. So be aware the
1591 fix you are hoping for might be one of those that won't be backported to the
1592 version line your care about. In that case you'll have no other choice then to
1593 live with the issue or switch to a newer Linux version, unless you want to
1594 patch the fix into your kernels yourself.
1599 *Perform the first three steps in the section "Reporting issues only
1600 occurring in older kernel version lines" above.*
1602 You need to carry out a few steps already described in another section of this
1603 guide. Those steps will let you:
1605 * Check if the kernel developers still maintain the Linux kernel version line
1608 * Search the Linux stable mailing list for exiting reports.
1610 * Check with the latest release.
1613 Check code history and search for existing discussions
1614 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1616 *Search the Linux kernel version control system for the change that fixed
1617 the issue in mainline, as its commit message might tell you if the fix is
1618 scheduled for backporting already. If you don't find anything that way,
1619 search the appropriate mailing lists for posts that discuss such an issue
1620 or peer-review possible fixes; then check the discussions if the fix was
1621 deemed unsuitable for backporting. If backporting was not considered at
1622 all, join the newest discussion, asking if it's in the cards.*
1624 In a lot of cases the issue you deal with will have happened with mainline, but
1625 got fixed there. The commit that fixed it would need to get backported as well
1626 to get the issue solved. That's why you want to search for it or any
1627 discussions abound it.
1629 * First try to find the fix in the Git repository that holds the Linux kernel
1630 sources. You can do this with the web interfaces `on kernel.org
1631 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_
1632 or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have
1633 a local clone you alternatively can search on the command line with ``git
1634 log --grep=<pattern>``.
1636 If you find the fix, look if the commit message near the end contains a
1637 'stable tag' that looks like this:
1639 Cc: <stable@vger.kernel.org> # 5.4+
1641 If that's case the developer marked the fix safe for backporting to version
1642 line 5.4 and later. Most of the time it's getting applied there within two
1643 weeks, but sometimes it takes a bit longer.
1645 * If the commit doesn't tell you anything or if you can't find the fix, look
1646 again for discussions about the issue. Search the net with your favorite
1647 internet search engine as well as the archives for the `Linux kernel
1648 developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the
1649 section `Locate kernel area that causes the issue` above and follow the
1650 instructions to find the subsystem in question: its bug tracker or mailing
1651 list archive might have the answer you are looking for.
1653 * If you see a proposed fix, search for it in the version control system as
1654 outlined above, as the commit might tell you if a backport can be expected.
1656 * Check the discussions for any indicators the fix might be too risky to get
1657 backported to the version line you care about. If that's the case you have
1658 to live with the issue or switch to the kernel version line where the fix
1661 * If the fix doesn't contain a stable tag and backporting was not discussed,
1662 join the discussion: mention the version where you face the issue and that
1663 you would like to see it fixed, if suitable.
1669 *One of the former steps should lead to a solution. If that doesn't work
1670 out, ask the maintainers for the subsystem that seems to be causing the
1671 issue for advice; CC the mailing list for the particular subsystem as well
1672 as the stable mailing list.*
1674 If the previous three steps didn't get you closer to a solution there is only
1675 one option left: ask for advice. Do that in a mail you sent to the maintainers
1676 for the subsystem where the issue seems to have its roots; CC the mailing list
1677 for the subsystem as well as the stable mailing list (stable@vger.kernel.org).
1680 Why some issues won't get any reaction or remain unfixed after being reported
1681 =============================================================================
1683 When reporting a problem to the Linux developers, be aware only 'issues of high
1684 priority' (regressions, security issues, severe problems) are definitely going
1685 to get resolved. The maintainers or if all else fails Linus Torvalds himself
1686 will make sure of that. They and the other kernel developers will fix a lot of
1687 other issues as well. But be aware that sometimes they can't or won't help; and
1688 sometimes there isn't even anyone to send a report to.
1690 This is best explained with kernel developers that contribute to the Linux
1691 kernel in their spare time. Quite a few of the drivers in the kernel were
1692 written by such programmers, often because they simply wanted to make their
1693 hardware usable on their favorite operating system.
1695 These programmers most of the time will happily fix problems other people
1696 report. But nobody can force them to do, as they are contributing voluntarily.
1698 Then there are situations where such developers really want to fix an issue,
1699 but can't: sometimes they lack hardware programming documentation to do so.
1700 This often happens when the publicly available docs are superficial or the
1701 driver was written with the help of reverse engineering.
1703 Sooner or later spare time developers will also stop caring for the driver.
1704 Maybe their test hardware broke, got replaced by something more fancy, or is so
1705 old that it's something you don't find much outside of computer museums
1706 anymore. Sometimes developer stops caring for their code and Linux at all, as
1707 something different in their life became way more important. In some cases
1708 nobody is willing to take over the job as maintainer – and nobody can be forced
1709 to, as contributing to the Linux kernel is done on a voluntary basis. Abandoned
1710 drivers nevertheless remain in the kernel: they are still useful for people and
1711 removing would be a regression.
1713 The situation is not that different with developers that are paid for their
1714 work on the Linux kernel. Those contribute most changes these days. But their
1715 employers sooner or later also stop caring for their code or make its
1716 programmer focus on other things. Hardware vendors for example earn their money
1717 mainly by selling new hardware; quite a few of them hence are not investing
1718 much time and energy in maintaining a Linux kernel driver for something they
1719 stopped selling years ago. Enterprise Linux distributors often care for a
1720 longer time period, but in new versions often leave support for old and rare
1721 hardware aside to limit the scope. Often spare time contributors take over once
1722 a company orphans some code, but as mentioned above: sooner or later they will
1723 leave the code behind, too.
1725 Priorities are another reason why some issues are not fixed, as maintainers
1726 quite often are forced to set those, as time to work on Linux is limited.
1727 That's true for spare time or the time employers grant their developers to
1728 spend on maintenance work on the upstream kernel. Sometimes maintainers also
1729 get overwhelmed with reports, even if a driver is working nearly perfectly. To
1730 not get completely stuck, the programmer thus might have no other choice than
1731 to prioritize issue reports and reject some of them.
1733 But don't worry too much about all of this, a lot of drivers have active
1734 maintainers who are quite interested in fixing as many issues as possible.
1740 Compared with other Free/Libre & Open Source Software it's hard to report
1741 issues to the Linux kernel developers: the length and complexity of this
1742 document and the implications between the lines illustrate that. But that's how
1743 it is for now. The main author of this text hopes documenting the state of the
1744 art will lay some groundwork to improve the situation over time.
1748 This text is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If you
1749 spot a typo or small mistake, feel free to let him know directly and he'll
1750 fix it. You are free to do the same in a mostly informal way if you want
1751 to contribute changes to the text, but for copyright reasons please CC
1752 linux-doc@vger.kernel.org and "sign-off" your contribution as
1753 Documentation/process/submitting-patches.rst outlines in the section "Sign
1754 your work - the Developer's Certificate of Origin".