How to read the MAINTAINERS file¶

To illustrate how to use theMAINTAINERS file, let’s assumethe WiFi in your Laptop suddenly misbehaves after updating the kernel. In thatcase it’s likely an issue in the WiFi driver. Obviously it could also be somecode it builds upon, but unless you suspect something like that stick to thedriver. If it’s really something else, the driver’s developers will get theright people involved.

Sadly, there is no way to check which code is driving a particular hardwarecomponent that is both universal and easy.

In case of a problem with the WiFi driver you for example might want to look atthe output oflspci-k, as it lists devices on the PCI/PCIe bus and thekernel module driving it:

[user@something ~]$ lspci -k[...]3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)  Subsystem: Bigfoot Networks, Inc. Device 1535  Kernel driver in use: ath10k_pci  Kernel modules: ath10k_pci[...]

But this approach won’t work if your WiFi chip is connected over USB or someother internal bus. In those cases you might want to check your WiFi manager orthe output ofiplink. Look for the name of the problematic networkinterface, which might be something like ‘wlp58s0’. This name can be used likethis to find the module driving it:

[user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/moduleath10k_pci

In case tricks like these don’t bring you any further, try to search theinternet on how to narrow down the driver or subsystem in question. And if youare unsure which it is: just try your best guess, somebody will help you if youguessed poorly.

Once you know the driver or subsystem, you want to search for it in theMAINTAINERS file. In the case of ‘ath10k_pci’ you won’t find anything, as thename is too specific. Sometimes you will need to search on the net for help;but before doing so, try a somewhat shorted or modified name when searching theMAINTAINERS file, as then you might find something like this:

QUALCOMM ATHEROS ATH10K WIRELESS DRIVERMail:          A. Some Human <shuman@example.com>Mailing list:  ath10k@lists.infradead.orgStatus:        SupportedWeb-page:      https://wireless.wiki.kernel.org/en/users/Drivers/ath10kSCM:           git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.gitFiles:         drivers/net/wireless/ath/ath10k/

Note: the line description will be abbreviations, if you read the plainMAINTAINERS file found in the root of the Linux source tree. ‘Mail:’ forexample will be ‘M:’, ‘Mailing list:’ will be ‘L’, and ‘Status:’ will be ‘S:’.A section near the top of the file explains these and other abbreviations.

First look at the line ‘Status’. Ideally it should be ‘Supported’ or‘Maintained’. If it states ‘Obsolete’ then you are using some outdated approachthat was replaced by a newer solution you need to switch to. Sometimes the codeonly has someone who provides ‘Odd Fixes’ when feeling motivated. And with‘Orphan’ you are totally out of luck, as nobody takes care of the code anymore.That only leaves these options: arrange yourself to live with the issue, fix ityourself, or find a programmer somewhere willing to fix it.

After checking the status, look for a line starting with ‘bugs:’: it will tellyou where to find a subsystem specific bug tracker to file your issue. Theexample above does not have such a line. That is the case for most sections, asLinux kernel development is completely driven by mail. Very few subsystems usea bug tracker, and only some of those rely on bugzilla.kernel.org.

In this and many other cases you thus have to look for lines starting with‘Mail:’ instead. Those mention the name and the email addresses for themaintainers of the particular code. Also look for a line starting with ‘Mailinglist:’, which tells you the public mailing list where the code is developed.Your report later needs to go by mail to those addresses. Additionally, for allissue reports sent by email, make sure to add the Linux Kernel Mailing List(LKML) <linux-kernel@vger.kernel.org> to CC. Don’t omit either of the mailinglists when sending your issue report by mail later! Maintainers are busy peopleand might leave some work for other developers on the subsystem specific list;and LKML is important to have one place where all issue reports can be found.

Finding the maintainers with the help of a script¶

For people that have the Linux sources at hand there is a second option to findthe proper place to report: the script ‘scripts/get_maintainer.pl’ which triesto find all people to contact. It queries the MAINTAINERS file and needs to becalled with a path to the source code in question. For drivers compiled asmodule if often can be found with a command like this:

$ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!'drivers/net/wireless/ath/ath10k/ath10k_pci.ko

Pass parts of this to the script:

$ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k*Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS)ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS))netdev@vger.kernel.org (open list:NETWORKING DRIVERS)linux-kernel@vger.kernel.org (open list)

Don’t sent your report to all of them. Send it to the maintainers, which thescript calls “supporter:”; additionally CC the most specific mailing list forthe code as well as the Linux Kernel Mailing List (LKML). In this case you thuswould need to send the report to ‘Some Human <shuman@example.com>’ with‘ath10k@lists.infradead.org’ and‘linux-kernel@vger.kernel.org’ in CC.

Note: in case you cloned the Linux sources with git you might want to callget_maintainer.pl a second time with--git. The script then will lookat the commit history to find which people recently worked on the code inquestion, as they might be able to help. But use these results with care, as itcan easily send you in a wrong direction. That for example happens quickly inareas rarely changed (like old or unmaintained drivers): sometimes such code ismodified during tree-wide cleanups by developers that do not care about theparticular driver at all.

Choosing the right version for testing¶

Head over tokernel.org to find out which version youwant to use for testing. Ignore the big yellow button that says ‘Latest release’and look a little lower at the table. At its top you’ll see a line starting withmainline, which most of the time will point to a pre-release with a versionnumber like ‘5.8-rc2’. If that’s the case, you’ll want to use this mainlinekernel for testing, as that where all fixes have to be applied first. Do not letthat ‘rc’ scare you, these ‘development kernels’ are pretty reliable — and youmade a backup, as you were instructed above, didn’t you?

In about two out of every nine to ten weeks, mainline might point you to aproper release with a version number like ‘5.7’. If that happens, considersuspending the reporting process until the first pre-release of the nextversion (5.8-rc1) shows up on kernel.org. That’s because the Linux developmentcycle then is in its two-week long ‘merge window’. The bulk of the changes andall intrusive ones get merged for the next release during this time. It’s a bitmore risky to use mainline during this period. Kernel developers are also oftenquite busy then and might have no spare time to deal with issue reports. It’salso quite possible that one of the many changes applied during the mergewindow fixes the issue you face; that’s why you soon would have to retest witha newer kernel version anyway, as outlined below in the section ‘Duties afterthe report went out’.

That’s why it might make sense to wait till the merge window is over. But don’tto that if you’re dealing with something that shouldn’t wait. In that caseconsider obtaining the latest mainline kernel via git (see below) or use thelatest stable version offered on kernel.org. Using that is also acceptable incase mainline for some reason does currently not work for you. An in general:using it for reproducing the issue is also better than not reporting it issueat all.

Better avoid using the latest stable kernel outside merge windows, as all fixesmust be applied to mainline first. That’s why checking the latest mainlinekernel is so important: any issue you want to see fixed in older version linesneeds to be fixed in mainline first before it can get backported, which cantake a few days or weeks. Another reason: the fix you hope for might be toohard or risky for backporting; reporting the issue again hence is unlikely tochange anything.

These aspects are also why longterm kernels (sometimes called “LTS kernels”)are unsuitable for this part of the reporting process: they are to distant fromthe current code. Hence go and test mainline first and follow the processfurther: if the issue doesn’t occur with mainline it will guide you how to getit fixed in older version lines, if that’s in the cards for the fix in question.

How to obtain a fresh Linux kernel¶

Using a pre-compiled kernel: This is often the quickest, easiest, and safestway for testing — especially is you are unfamiliar with the Linux kernel. Theproblem: most of those shipped by distributors or add-on repositories are buildfrom modified Linux sources. They are thus not vanilla and therefore oftenunsuitable for testing and issue reporting: the changes might cause the issueyou face or influence it somehow.

But you are in luck if you are using a popular Linux distribution: for quite afew of them you’ll find repositories on the net that contain packages with thelatest mainline or stable Linux built as vanilla kernel. It’s totally okay touse these, just make sure from the repository’s description they are vanilla orat least close to it. Additionally ensure the packages contain the latestversions as offered on kernel.org. The packages are likely unsuitable if theyare older than a week, as new mainline and stable kernels typically get releasedat least once a week.

Please note that you might need to build your own kernel manually later: that’ssometimes needed for debugging or testing fixes, as described later in thisdocument. Also be aware that pre-compiled kernels might lack debug symbols thatare needed to decode messages the kernel prints when a panic, Oops, warning, orBUG occurs; if you plan to decode those, you might be better off compiling akernel yourself (see the end of this subsection and the section titled ‘Decodefailure messages’ for details).

Using git: Developers and experienced Linux users familiar with git areoften best served by obtaining the latest Linux kernel sources straight from theofficial development repository on kernel.org.Those are likely a bit ahead of the latest mainline pre-release. Don’t worryabout it: they are as reliable as a proper pre-release, unless the kernel’sdevelopment cycle is currently in the middle of a merge window. But even thenthey are quite reliable.

Conventional: People unfamiliar with git are often best served bydownloading the sources as tarball fromkernel.org.

How to actually build a kernel is not described here, as many websites explainthe necessary steps already. If you are new to it, consider following one ofthose how-to’s that suggest to usemakelocalmodconfig, as that tries topick up the configuration of your current kernel and then tries to adjust itsomewhat for your system. That does not make the resulting kernel any better,but quicker to compile.

Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel,please try to enable CONFIG_KALLSYMS when configuring your kernel.Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; thelatter is the relevant one of those two, but can only be reached if you enablethe former. Be aware CONFIG_DEBUG_INFO increases the storage space required tobuild a kernel by quite a bit. But that’s worth it, as these options will allowyou later to pinpoint the exact line of code that triggers your issue. Thesection ‘Decode failure messages’ below explains this in more detail.

But keep in mind: Always keep a record of the issue encountered in case it ishard to reproduce. Sending an undecoded report is better than not reportingthe issue at all.

Things each report should mention¶

Describe in detail how your issue happens with the fresh vanilla kernel youinstalled. Try to include the step-by-step instructions you wrote and optimizedearlier that outline how you and ideally others can reproduce the issue; inthose rare cases where that’s impossible try to describe what you did totrigger it.

Also include all the relevant information others might need to understand theissue and its environment. What’s actually needed depends a lot on the issue,but there are some things you should include always:

• the output fromcat/proc/version, which contains the Linux kernelversion number and the compiler it was built with.

• the Linux distribution the machine is running (hostnamectl|grep"OperatingSystem")

• the architecture of the CPU and the operating system (uname-mi)

• if you are dealing with a regression and performed a bisection, mention thesubject and the commit-id of the change that is causing it.

In a lot of cases it’s also wise to make two more things available to thosethat read your report:

• the configuration used for building your Linux kernel (the ‘.config’ file)

• the kernel’s messages that you get fromdmesg written to a file. Makesure that it starts with a line like ‘Linux version 5.8-1(foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug3 14:54:37 UTC 2020’ If it’s missing, then important messages from the firstboot phase already got discarded. In this case instead consider usingjournalctl-b0-k; alternatively you can also reboot, reproduce theissue and calldmesg right afterwards.

These two files are big, that’s why it’s a bad idea to put them directly intoyour report. If you are filing the issue in a bug tracker then attach them tothe ticket. If you report the issue by mail do not attach them, as that makesthe mail too large; instead do one of these things:

• Upload the files somewhere public (your website, a public file pasteservice, a ticket created just for this purpose onbugzilla.kernel.org, ...) and include a link to them in yourreport. Ideally use something where the files stay available for years, asthey could be useful to someone many years from now; this for example canhappen if five or ten years from now a developer works on some code that waschanged just to fix your issue.

• Put the files aside and mention you will send them later in individualreplies to your own mail. Just remember to actually do that once the reportwent out. ;-)

Things that might be wise to provide¶

Depending on the issue you might need to add more background data. Here are afew suggestions what often is good to provide:

• If you are dealing with a ‘warning’, an ‘OOPS’ or a ‘panic’ from the kernel,include it. If you can’t copy’n’paste it, try to capture a netconsole traceor at least take a picture of the screen.

• If the issue might be related to your computer hardware, mention what kindof system you use. If you for example have problems with your graphics card,mention its manufacturer, the card’s model, and what chip is uses. If it’s alaptop mention its name, but try to make sure it’s meaningful. ‘Dell XPS 13’for example is not, because it might be the one from 2012; that one looksnot that different from the one sold today, but apart from that the two havenothing in common. Hence, in such cases add the exact model number, whichfor example are ‘9380’ or ‘7390’ for XPS 13 models introduced during 2019.Names like ‘Lenovo Thinkpad T590’ are also somewhat ambiguous: there arevariants of this laptop with and without a dedicated graphics chip, so tryto find the exact model name or specify the main components.

• Mention the relevant software in use. If you have problems with loadingmodules, you want to mention the versions of kmod, systemd, and udev in use.If one of the DRM drivers misbehaves, you want to state the versions oflibdrm and Mesa; also specify your Wayland compositor or the X-Server andits driver. If you have a filesystem issue, mention the version ofcorresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...).

• Gather additional information from the kernel that might be of interest. Theoutput fromlspci-nn will for example help others to identify whathardware you use. If you have a problem with hardware you even might want tomake the output fromsudolspci-vvv available, as that providesinsights how the components were configured. For some issues it might begood to include the contents of files like/proc/cpuinfo,/proc/ioports,/proc/iomem,/proc/modules, or/proc/scsi/scsi. Some subsystem also offer tools to collect relevantinformation. One such tool isalsa-info.shwhich the audio/soundsubsystem developers provide.

Those examples should give your some ideas of what data might be wise toattach, but you have to think yourself what will be helpful for others to know.Don’t worry too much about forgetting something, as developers will ask foradditional details they need. But making everything important available fromthe start increases the chance someone will take a closer look.

The important part: the head of your report¶

Now that you have the detailed part of the report prepared let’s get to themost important section: the first few sentences. Thus go to the top, addsomething like ‘The detailed description:’ before the part you just wrote andinsert two newlines at the top. Now write one normal length paragraph thatdescribes the issue roughly. Leave out all boring details and focus on thecrucial parts readers need to know to understand what this is all about; if youthink this bug affects a lot of users, mention this to get people interested.

Once you did that insert two more lines at the top and write a one sentencesummary that explains quickly what the report is about. After that you have toget even more abstract and write an even shorter subject/title for the report.

Now that you have written this part take some time to optimize it, as it is themost important parts of your report: a lot of people will only read this beforethey decide if reading the rest is time well spent.

Now send or file the report like theMAINTAINERS file toldyou, unless it’s one of those ‘issues of high priority’ outlined earlier: inthat case please read the next subsection first before sending the report onits way.

Special handling for high priority issues¶

Reports for high priority issues need special handling.

Severe issues: make sure the subject or ticket title as well as the firstparagraph makes the severeness obvious.

Regressions: make the report’s subject start with ‘[REGRESSION]’.

In case you performed a successful bisection, use the title of the change thatintroduced the regression as the second part of your subject. Make the reportalso mention the commit id of the culprit. In case of an unsuccessful bisection,make your report mention the latest tested version that’s working fine (say 5.7)and the oldest where the issue occurs (say 5.8-rc1).

When sending the report by mail, CC the Linux regressions mailing list(regressions@lists.linux.dev). In case the report needs to be filed to some webtracker, proceed to do so. Once filed, forward the report by mail to theregressions list; CC the maintainer and the mailing list for the subsystem inquestion. Make sure to inline the forwarded report, hence do not attach it.Also add a short note at the top where you mention the URL to the ticket.

When mailing or forwarding the report, in case of a successful bisection add theauthor of the culprit to the recipients; also CC everyone in the signed-off-bychain, which you find at the end of its commit message.

Security issues: for these issues your will have to evaluate if ashort-term risk to other users would arise if details were publicly disclosed.If that’s not the case simply proceed with reporting the issue as described.For issues that bear such a risk you will need to adjust the reporting processslightly:

• If the MAINTAINERS file instructed you to report the issue by mail, do notCC any public mailing lists.

• If you were supposed to file the issue in a bug tracker make sure to markthe ticket as ‘private’ or ‘security issue’. If the bug tracker does notoffer a way to keep reports private, forget about it and send your report asa private mail to the maintainers instead.

In both cases make sure to also mail your report to the addresses theMAINTAINERS file lists in the section ‘security contact’. Ideally directly CCthem when sending the report by mail. If you filed it in a bug tracker, forwardthe report’s text to these addresses; but on top of it put a small note whereyou mention that you filed it with a link to the ticket.

SeeSecurity bugs for more information.

General advice for further interactions¶

Always reply in public: When you filed the issue in a bug tracker, alwaysreply there and do not contact any of the developers privately about it. Formailed reports always use the ‘Reply-all’ function when replying to any mailsyou receive. That includes mails with any additional data you might want to addto your report: go to your mail applications ‘Sent’ folder and use ‘reply-all’on your mail with the report. This approach will make sure the public mailinglist(s) and everyone else that gets involved over time stays in the loop; italso keeps the mail thread intact, which among others is really important formailing lists to group all related mails together.

There are just two situations where a comment in a bug tracker or a ‘Reply-all’is unsuitable:

• Someone tells you to send something privately.

• You were told to send something, but noticed it contains sensitiveinformation that needs to be kept private. In that case it’s okay to send itin private to the developer that asked for it. But note in the ticket or amail that you did that, so everyone else knows you honored the request.

Do research before asking for clarifications or help: In this part of theprocess someone might tell you to do something that requires a skill you mightnot have mastered yet. For example, you might be asked to use some test toolsyou never have heard of yet; or you might be asked to apply a patch to theLinux kernel sources to test if it helps. In some cases it will be fine sendinga reply asking for instructions how to do that. But before going that route tryto find the answer own your own by searching the internet; alternativelyconsider asking in other places for advice. For example ask a friend or postabout it to a chatroom or forum you normally hang out.

Be patient: If you are really lucky you might get a reply to your reportwithin a few hours. But most of the time it will take longer, as maintainersare scattered around the globe and thus might be in a different time zone – onewhere they already enjoy their night away from keyboard.

In general, kernel developers will take one to five business days to respond toreports. Sometimes it will take longer, as they might be busy with the mergewindows, other work, visiting developer conferences, or simply enjoying a longsummer holiday.

The ‘issues of high priority’ (see above for an explanation) are an exceptionhere: maintainers should address them as soon as possible; that’s why youshould wait a week at maximum (or just two days if it’s something urgent)before sending a friendly reminder.

Sometimes the maintainer might not be responding in a timely manner; othertimes there might be disagreements, for example if an issue qualifies asregression or not. In such cases raise your concerns on the mailing list andask others for public or private replies how to move on. If that fails, itmight be appropriate to get a higher authority involved. In case of a WiFidriver that would be the wireless maintainers; if there are no higher levelmaintainers or all else fails, it might be one of those rare situations whereit’s okay to get Linus Torvalds involved.

Proactive testing: Every time the first pre-release (the ‘rc1’) of a newmainline kernel version gets released, go and check if the issue is fixed thereor if anything of importance changed. Mention the outcome in the ticket or in amail you sent as reply to your report (make sure it has all those in the CCthat up to that point participated in the discussion). This will show yourcommitment and that you are willing to help. It also tells developers if theissue persists and makes sure they do not forget about it. A few otheroccasional retests (for example with rc3, rc5 and the final) are also a goodidea, but only report your results if something relevant changed or if you arewriting something anyway.

With all these general things off the table let’s get into the details of howto help to get issues resolved once they were reported.

Inquires and testing request¶

Here are your duties in case you got replies to your report:

Check who you deal with: Most of the time it will be the maintainer or adeveloper of the particular code area that will respond to your report. But asissues are normally reported in public it could be anyone that’s replying —including people that want to help, but in the end might guide you totally offtrack with their questions or requests. That rarely happens, but it’s one ofmany reasons why it’s wise to quickly run an internet search to see who you’reinteracting with. By doing this you also get aware if your report was heard bythe right people, as a reminder to the maintainer (see below) might be in orderlater if discussion fades out without leading to a satisfying solution for theissue.

Inquiries for data: Often you will be asked to test something or provideadditional details. Try to provide the requested information soon, as you havethe attention of someone that might help and risk losing it the longer youwait; that outcome is even likely if you do not provide the information withina few business days.

Requests for testing: When you are asked to test a diagnostic patch or apossible fix, try to test it in timely manner, too. But do it properly and makesure to not rush it: mixing things up can happen easily and can lead to a lotof confusion for everyone involved. A common mistake for example is thinking aproposed patch with a fix was applied, but in fact wasn’t. Things like thathappen even to experienced testers occasionally, but they most of the time willnotice when the kernel with the fix behaves just as one without it.

What to do when nothing of substance happens¶

Some reports will not get any reaction from the responsible Linux kerneldevelopers; or a discussion around the issue evolved, but faded out withnothing of substance coming out of it.

In these cases wait two (better: three) weeks before sending a friendlyreminder: maybe the maintainer was just away from keyboard for a while whenyour report arrived or had something more important to take care of. Whenwriting the reminder, kindly ask if anything else from your side is needed toget the ball running somehow. If the report got out by mail, do that in thefirst lines of a mail that is a reply to your initial mail (see above) whichincludes a full quote of the original report below: that’s on of those fewsituations where such a ‘TOFU’ (Text Over, Fullquote Under) is the rightapproach, as then all the recipients will have the details at hand immediatelyin the proper order.

After the reminder wait three more weeks for replies. If you still don’t get aproper reaction, you first should reconsider your approach. Did you maybe tryto reach out to the wrong people? Was the report maybe offensive or soconfusing that people decided to completely stay away from it? The best way torule out such factors: show the report to one or two people familiar with FLOSSissue reporting and ask for their opinion. Also ask them for their advice howto move forward. That might mean: prepare a better report and make those peoplereview it before you send it out. Such an approach is totally fine; justmention that this is the second and improved report on the issue and include alink to the first report.

If the report was proper you can send a second reminder; in it ask for advicewhy the report did not get any replies. A good moment for this second remindermail is shortly after the first pre-release (the ‘rc1’) of a new Linux kernelversion got published, as you should retest and provide a status update at thatpoint anyway (see above).

If the second reminder again results in no reaction within a week, try tocontact a higher-level maintainer asking for advice: even busy maintainers bythen should at least have sent some kind of acknowledgment.

Remember to prepare yourself for a disappointment: maintainers ideally shouldreact somehow to every issue report, but they are only obliged to fix those‘issues of high priority’ outlined earlier. So don’t be too devastating if youget a reply along the lines of ‘thanks for the report, I have more importantissues to deal with currently and won’t have time to look into this for theforeseeable future’.

It’s also possible that after some discussion in the bug tracker or on a listnothing happens anymore and reminders don’t help to motivate anyone to work outa fix. Such situations can be devastating, but is within the cards when itcomes to Linux kernel development. This and several other reasons for notgetting help are explained in ‘Why some issues won’t get any reaction or remainunfixed after being reported’ near the end of this document.

Don’t get devastated if you don’t find any help or if the issue in the end doesnot get solved: the Linux kernel is FLOSS and thus you can still help yourself.You for example could try to find others that are affected and team up withthem to get the issue resolved. Such a team could prepare a fresh reporttogether that mentions how many you are and why this is something that in youroption should get fixed. Maybe together you can also narrow down the root causeor the change that introduced a regression, which often makes developing a fixeasier. And with a bit of luck there might be someone in the team that knows abit about programming and might be able to write a fix.

Make sure the particular version line still gets support¶

Check if the kernel developers still maintain the Linux kernel versionline you care about: go to the front page of kernel.org and make sure itmentions the latest release of the particular version line without an‘[EOL]’ tag.

Most kernel version lines only get supported for about three months, asmaintaining them longer is quite a lot of work. Hence, only one per year ischosen and gets supported for at least two years (often six). That’s why youneed to check if the kernel developers still support the version line you carefor.

Note, if kernel.org lists two stable version lines on the front page, youshould consider switching to the newer one and forget about the older one:support for it is likely to be abandoned soon. Then it will get a “end-of-life”(EOL) stamp. Version lines that reached that point still get mentioned on thekernel.org front page for a week or two, but are unsuitable for testing andreporting.

Search stable mailing list¶

Check the archives of the Linux stable mailing list for existing reports.

Maybe the issue you face is already known and was fixed or is about to. Hence,search the archives of the Linux stable mailing list for reports about an issue like yours. Ifyou find any matches, consider joining the discussion, unless the fix isalready finished and scheduled to get applied soon.

Reproduce issue with the newest release¶

Install the latest release from the particular version line as a vanillakernel. Ensure this kernel is not tainted and still shows the problem, asthe issue might have already been fixed there. If you first noticed theproblem with a vendor kernel, check a vanilla build of the last versionknown to work performs fine as well.

Before investing any more time in this process you want to check if the issuewas already fixed in the latest release of version line you’re interested in.This kernel needs to be vanilla and shouldn’t be tainted before the issuehappens, as detailed outlined already above in the section “Install a freshkernel for testing”.

Did you first notice the regression with a vendor kernel? Then changes thevendor applied might be interfering. You need to rule that out by performinga recheck. Say something broke when you updated from 5.10.4-vendor.42 to5.10.5-vendor.43. Then after testing the latest 5.10 release as outlined inthe previous paragraph check if a vanilla build of Linux 5.10.4 works fine aswell. If things are broken there, the issue does not qualify as upstreamregression and you need switch back to the main step-by-step guide to reportthe issue.

Report the regression¶

Send a short problem report to the Linux stable mailing list(stable@vger.kernel.org) and CC the Linux regressions mailing list(regressions@lists.linux.dev); if you suspect the cause in a particularsubsystem, CC its maintainer and its mailing list. Roughly describe theissue and ideally explain how to reproduce it. Mention the first versionthat shows the problem and the last version that’s working fine. Thenwait for further instructions.

When reporting a regression that happens within a stable or longterm kernelline (say when updating from 5.10.4 to 5.10.5) a brief report is enough forthe start to get the issue reported quickly. Hence a rough description to thestable and regressions mailing list is all it takes; but in case you suspectthe cause in a particular subsystem, CC its maintainers and its mailing listas well, because that will speed things up.

And note, it helps developers a great deal if you can specify the exact versionthat introduced the problem. Hence if possible within a reasonable time frame,try to find that version using vanilla kernels. Let’s assume something broke whenyour distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then asinstructed above go and check the latest kernel from that version line, say5.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patchesthe distributor applied interfere. If the issue doesn’t manifest itself there,try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find thefirst version where things broke. Mention it in the report and state that 5.10.9is still broken.

What the previous paragraph outlines is basically a rough manual ‘bisection’.Once your report is out your might get asked to do a proper one, as it allows topinpoint the exact change that causes the issue (which then can easily getreverted to fix the issue quickly). Hence consider to do a proper bisectionright away if time permits. See the section ‘Special care for regressions’ andthe documentBisecting a regression for details how toperform one. In case of a successful bisection add the author of the culprit tothe recipients; also CC everyone in the signed-off-by chain, which you find atthe end of its commit message.

Some fixes are too complex¶

Prepare yourself for the possibility that going through the next few stepsmight not get the issue solved in older releases: the fix might be too bigor risky to get backported there.

Even small and seemingly obvious code-changes sometimes introduce new andtotally unexpected problems. The maintainers of the stable and longterm kernelsare very aware of that and thus only apply changes to these kernels that arewithin rules outlined inEverything you ever wanted to know about Linux -stable releases.

Complex or risky changes for example do not qualify and thus only get appliedto mainline. Other fixes are easy to get backported to the newest stable andlongterm kernels, but too risky to integrate into older ones. So be aware thefix you are hoping for might be one of those that won’t be backported to theversion line your care about. In that case you’ll have no other choice then tolive with the issue or switch to a newer Linux version, unless you want topatch the fix into your kernels yourself.

Common preparations¶

Perform the first three steps in the section “Reporting issues onlyoccurring in older kernel version lines” above.

You need to carry out a few steps already described in another section of thisguide. Those steps will let you:

• Check if the kernel developers still maintain the Linux kernel version lineyou care about.

• Search the Linux stable mailing list for exiting reports.

• Check with the latest release.

Check code history and search for existing discussions¶

Search the Linux kernel version control system for the change that fixedthe issue in mainline, as its commit message might tell you if the fix isscheduled for backporting already. If you don’t find anything that way,search the appropriate mailing lists for posts that discuss such an issueor peer-review possible fixes; then check the discussions if the fix wasdeemed unsuitable for backporting. If backporting was not considered atall, join the newest discussion, asking if it’s in the cards.

In a lot of cases the issue you deal with will have happened with mainline, butgot fixed there. The commit that fixed it would need to get backported as wellto get the issue solved. That’s why you want to search for it or anydiscussions abound it.

• First try to find the fix in the Git repository that holds the Linux kernelsources. You can do this with the web interfaceson kernel.orgor its mirroron GitHub; if you havea local clone you alternatively can search on the command line withgitlog--grep=<pattern>.

  If you find the fix, look if the commit message near the end contains a‘stable tag’ that looks like this:

  Cc: <stable@vger.kernel.org> # 5.4+

  If that’s case the developer marked the fix safe for backporting to versionline 5.4 and later. Most of the time it’s getting applied there within twoweeks, but sometimes it takes a bit longer.

• If the commit doesn’t tell you anything or if you can’t find the fix, lookagain for discussions about the issue. Search the net with your favoriteinternet search engine as well as the archives for theLinux kerneldevelopers mailing list. Also read thesectionLocate kernel area that causes the issue above and follow theinstructions to find the subsystem in question: its bug tracker or mailinglist archive might have the answer you are looking for.

• If you see a proposed fix, search for it in the version control system asoutlined above, as the commit might tell you if a backport can be expected.

  • Check the discussions for any indicators the fix might be too risky to getbackported to the version line you care about. If that’s the case you haveto live with the issue or switch to the kernel version line where the fixgot applied.

  • If the fix doesn’t contain a stable tag and backporting was not discussed,join the discussion: mention the version where you face the issue and thatyou would like to see it fixed, if suitable.

Ask for advice¶

One of the former steps should lead to a solution. If that doesn’t workout, ask the maintainers for the subsystem that seems to be causing theissue for advice; CC the mailing list for the particular subsystem as wellas the stable mailing list.

If the previous three steps didn’t get you closer to a solution there is onlyone option left: ask for advice. Do that in a mail you sent to the maintainersfor the subsystem where the issue seems to have its roots; CC the mailing listfor the subsystem as well as the stable mailing list (stable@vger.kernel.org).