3.1.Specifying the problem¶

Like any engineering project, a successful kernel enhancement starts with aclear description of the problem to be solved. In some cases, this step iseasy: when a driver is needed for a specific piece of hardware, forexample. In others, though, it is tempting to confuse the real problemwith the proposed solution, and that can lead to difficulties.

Consider an example: some years ago, developers working with Linux audiosought a way to run applications without dropouts or other artifacts causedby excessive latency in the system. The solution they arrived at was akernel module intended to hook into the Linux Security Module (LSM)framework; this module could be configured to give specific applicationsaccess to the realtime scheduler. This module was implemented and sent tothe linux-kernel mailing list, where it immediately ran into problems.

To the audio developers, this security module was sufficient to solve theirimmediate problem. To the wider kernel community, though, it was seen as amisuse of the LSM framework (which is not intended to confer privilegesonto processes which they would not otherwise have) and a risk to systemstability. Their preferred solutions involved realtime scheduling accessvia the rlimit mechanism for the short term, and ongoing latency reductionwork in the long term.

The audio community, however, could not see past the particular solutionthey had implemented; they were unwilling to accept alternatives. Theresulting disagreement left those developers feeling disillusioned with theentire kernel development process; one of them went back to an audio listand posted this:

There are a number of very good Linux kernel developers, but theytend to get outshouted by a large crowd of arrogant fools. Tryingto communicate user requirements to these people is a waste oftime. They are much too “intelligent” to listen to lesser mortals.

(https://lwn.net/Articles/131776/).

The reality of the situation was different; the kernel developers were farmore concerned about system stability, long-term maintenance, and findingthe right solution to the problem than they were with a specific module.The moral of the story is to focus on the problem - not a specific solution- and to discuss it with the development community before investing in thecreation of a body of code.

So, when contemplating a kernel development project, one should obtainanswers to a short set of questions:

• What, exactly, is the problem which needs to be solved?

• Who are the users affected by this problem? Which use cases should thesolution address?

• How does the kernel fall short in addressing that problem now?

Only then does it make sense to start considering possible solutions.

3.2.Early discussion¶

When planning a kernel development project, it makes great sense to holddiscussions with the community before launching into implementation. Earlycommunication can save time and trouble in a number of ways:

• It may well be that the problem is addressed by the kernel in ways whichyou have not understood. The Linux kernel is large and has a number offeatures and capabilities which are not immediately obvious. Not allkernel capabilities are documented as well as one might like, and it iseasy to miss things. Your author has seen the posting of a completedriver which duplicated an existing driver that the new author had beenunaware of. Code which reinvents existing wheels is not only wasteful;it will also not be accepted into the mainline kernel.

• There may be elements of the proposed solution which will not beacceptable for mainline merging. It is better to find out aboutproblems like this before writing the code.

• It’s entirely possible that other developers have thought about theproblem; they may have ideas for a better solution, and may be willingto help in the creation of that solution.

Years of experience with the kernel development community have taught aclear lesson: kernel code which is designed and developed behind closeddoors invariably has problems which are only revealed when the code isreleased into the community. Sometimes these problems are severe,requiring months or years of effort before the code can be brought up tothe kernel community’s standards. Some examples include:

• The Devicescape network stack was designed and implemented forsingle-processor systems. It could not be merged into the mainlineuntil it was made suitable for multiprocessor systems. Retrofittinglocking and such into code is a difficult task; as a result, the mergingof this code (now called mac80211) was delayed for over a year.

• The Reiser4 filesystem included a number of capabilities which, in thecore kernel developers’ opinion, should have been implemented in thevirtual filesystem layer instead. It also included features which couldnot easily be implemented without exposing the system to user-causeddeadlocks. The late revelation of these problems - and refusal toaddress some of them - has caused Reiser4 to stay out of the mainlinekernel.

• The AppArmor security module made use of internal virtual filesystemdata structures in ways which were considered to be unsafe andunreliable. This concern (among others) kept AppArmor out of themainline for years.

In each of these cases, a great deal of pain and extra work could have beenavoided with some early discussion with the kernel developers.

3.3.Who do you talk to?¶

When developers decide to take their plans public, the next question willbe: where do we start? The answer is to find the right mailing list(s) andthe right maintainer. For mailing lists, the best approach is to look inthe MAINTAINERS file for a relevant place to post. If there is a suitablesubsystem list, posting there is often preferable to posting onlinux-kernel; you are more likely to reach developers with expertise in therelevant subsystem and the environment may be more supportive.

Finding maintainers can be a bit harder. Again, the MAINTAINERS file isthe place to start. That file tends to not always be up to date, though,and not all subsystems are represented there. The person listed in theMAINTAINERS file may, in fact, not be the person who is actually acting inthat role currently. So, when there is doubt about who to contact, auseful trick is to use git (and “git log” in particular) to see who iscurrently active within the subsystem of interest. Look at who is writingpatches, and who, if anybody, is attaching Signed-off-by lines to thosepatches. Those are the people who will be best placed to help with a newdevelopment project.

The task of finding the right maintainer is sometimes challenging enoughthat the kernel developers have added a script to ease the process:

.../scripts/get_maintainer.pl

This script will return the current maintainer(s) for a given file ordirectory when given the “-f” option. If passed a patch on thecommand line, it will list the maintainers who should probably receivecopies of the patch. This is the preferred way (unlike “-f” option) to get thelist of people to Cc for your patches. There are a number of optionsregulating how hard get_maintainer.pl will search for maintainers; please becareful about using the more aggressive options as you may end up includingdevelopers who have no real interest in the code you are modifying.

If all else fails, talking to Andrew Morton can be an effective way totrack down a maintainer for a specific piece of code.

3.4.When to post?¶

If possible, posting your plans during the early stages can only behelpful. Describe the problem being solved and any plans that have beenmade on how the implementation will be done. Any information you canprovide can help the development community provide useful input on theproject.

One discouraging thing which can happen at this stage is not a hostilereaction, but, instead, little or no reaction at all. The sad truth of thematter is (1) kernel developers tend to be busy, (2) there is no shortageof people with grand plans and little code (or even prospect of code) toback them up, and (3) nobody is obligated to review or comment on ideasposted by others. Beyond that, high-level designs often hide problemswhich are only revealed when somebody actually tries to implement thosedesigns; for that reason, kernel developers would rather see the code.

If a request-for-comments posting yields little in the way of comments, donot assume that it means there is no interest in the project.Unfortunately, you also cannot assume that there are no problems with youridea. The best thing to do in this situation is to proceed, keeping thecommunity informed as you go.

3.5.Getting official buy-in¶

If your work is being done in a corporate environment - as most Linuxkernel work is - you must, obviously, have permission from suitablyempowered managers before you can post your company’s plans or code to apublic mailing list. The posting of code which has not been cleared forrelease under a GPL-compatible license can be especially problematic; thesooner that a company’s management and legal staff can agree on the postingof a kernel development project, the better off everybody involved will be.

Some readers may be thinking at this point that their kernel work isintended to support a product which does not yet have an officiallyacknowledged existence. Revealing their employer’s plans on a publicmailing list may not be a viable option. In cases like this, it is worthconsidering whether the secrecy is really necessary; there is often no realneed to keep development plans behind closed doors.

That said, there are also cases where a company legitimately cannotdisclose its plans early in the development process. Companies withexperienced kernel developers may choose to proceed in an open-loop manneron the assumption that they will be able to avoid serious integrationproblems later. For companies without that sort of in-house expertise, thebest option is often to hire an outside developer to review the plans undera non-disclosure agreement. The Linux Foundation operates an NDA programdesigned to help with this sort of situation; more information can be foundat:

https://www.linuxfoundation.org/nda/

This kind of review is often enough to avoid serious problems later onwithout requiring public disclosure of the project.