Executive Summary¶

You think you want a stable kernel interface, but you really do not, andyou don’t even know it. What you want is a stable running driver, andyou get that only if your driver is in the main kernel tree. You alsoget lots of other good benefits if your driver is in the main kerneltree, all of which has made Linux into such a strong, stable, and matureoperating system which is the reason you are using it in the firstplace.

Intro¶

It’s only the odd person who wants to write a kernel driver that needsto worry about the in-kernel interfaces changing. For the majority ofthe world, they neither see this interface, nor do they care about it atall.

First off, I’m not going to addressany legal issues about closedsource, hidden source, binary blobs, source wrappers, or any other termthat describes kernel drivers that do not have their source codereleased under the GPL. Please consult a lawyer if you have any legalquestions, I’m a programmer and hence, I’m just going to be describingthe technical issues here (not to make light of the legal issues, theyare real, and you do need to be aware of them at all times.)

So, there are two main topics here, binary kernel interfaces and stablekernel source interfaces. They both depend on each other, but we willdiscuss the binary stuff first to get it out of the way.

Binary Kernel Interface¶

Assuming that we had a stable kernel source interface for the kernel, abinary interface would naturally happen too, right? Wrong. Pleaseconsider the following facts about the Linux kernel:

• Depending on the version of the C compiler you use, different kerneldata structures will contain different alignment of structures, andpossibly include different functions in different ways (puttingfunctions inline or not.) The individual function organizationisn’t that important, but the different data structure padding isvery important.

• Depending on what kernel build options you select, a wide range ofdifferent things can be assumed by the kernel:

  • different structures can contain different fields

  • Some functions may not be implemented at all, (i.e. some lockscompile away to nothing for non-SMP builds.)

  • Memory within the kernel can be aligned in different ways,depending on the build options.

• Linux runs on a wide range of different processor architectures.There is no way that binary drivers from one architecture will runon another architecture properly.

Now a number of these issues can be addressed by simply compiling yourmodule for the exact specific kernel configuration, using the same exactC compiler that the kernel was built with. This is sufficient if youwant to provide a module for a specific release version of a specificLinux distribution. But multiply that single build by the number ofdifferent Linux distributions and the number of different supportedreleases of the Linux distribution and you quickly have a nightmare ofdifferent build options on different releases. Also realize that eachLinux distribution release contains a number of different kernels, alltuned to different hardware types (different processor types anddifferent options), so for even a single release you will need to createmultiple versions of your module.

Trust me, you will go insane over time if you try to support this kindof release, I learned this the hard way a long time ago...

Stable Kernel Source Interfaces¶

This is a much more “volatile” topic if you talk to people who try tokeep a Linux kernel driver that is not in the main kernel tree up todate over time.

Linux kernel development is continuous and at a rapid pace, neverstopping to slow down. As such, the kernel developers find bugs incurrent interfaces, or figure out a better way to do things. If they dothat, they then fix the current interfaces to work better. When they doso, function names may change, structures may grow or shrink, andfunction parameters may be reworked. If this happens, all of theinstances of where this interface is used within the kernel are fixed upat the same time, ensuring that everything continues to work properly.

As a specific examples of this, the in-kernel USB interfaces haveundergone at least three different reworks over the lifetime of thissubsystem. These reworks were done to address a number of differentissues:

• A change from a synchronous model of data streams to an asynchronousone. This reduced the complexity of a number of drivers andincreased the throughput of all USB drivers such that we are nowrunning almost all USB devices at their maximum speed possible.

• A change was made in the way data packets were allocated from theUSB core by USB drivers so that all drivers now needed to providemore information to the USB core to fix a number of documenteddeadlocks.

This is in stark contrast to a number of closed source operating systemswhich have had to maintain their older USB interfaces over time. Thisprovides the ability for new developers to accidentally use the oldinterfaces and do things in improper ways, causing the stability of theoperating system to suffer.

In both of these instances, all developers agreed that these wereimportant changes that needed to be made, and they were made, withrelatively little pain. If Linux had to ensure that it will preserve astable source interface, a new interface would have been created, andthe older, broken one would have had to be maintained over time, leadingto extra work for the USB developers. Since all Linux USB developers dotheir work on their own time, asking programmers to do extra work for nogain, for free, is not a possibility.

Security issues are also very important for Linux. When asecurity issue is found, it is fixed in a very short amount of time. Anumber of times this has caused internal kernel interfaces to bereworked to prevent the security problem from occurring. When thishappens, all drivers that use the interfaces were also fixed at thesame time, ensuring that the security problem was fixed and could notcome back at some future time accidentally. If the internal interfaceswere not allowed to change, fixing this kind of security problem andinsuring that it could not happen again would not be possible.

Kernel interfaces are cleaned up over time. If there is no one using acurrent interface, it is deleted. This ensures that the kernel remainsas small as possible, and that all potential interfaces are tested aswell as they can be (unused interfaces are pretty much impossible totest for validity.)

What to do¶

So, if you have a Linux kernel driver that is not in the main kerneltree, what are you, a developer, supposed to do? Releasing a binarydriver for every different kernel version for every distribution is anightmare, and trying to keep up with an ever changing kernel interfaceis also a rough job.

Simple, get your kernel driver into the main kernel tree (remember we aretalking about drivers released under a GPL-compatible license here, if yourcode doesn’t fall under this category, good luck, you are on your own here,you leech). If your driver is in the tree, and a kernel interface changes,it will be fixed up by the person who did the kernel change in the firstplace. This ensures that your driver is always buildable, and works overtime, with very little effort on your part.

The very good side effects of having your driver in the main kernel treeare:

• The quality of the driver will rise as the maintenance costs (to theoriginal developer) will decrease.

• Other developers will add features to your driver.

• Other people will find and fix bugs in your driver.

• Other people will find tuning opportunities in your driver.

• Other people will update the driver for you when external interfacechanges require it.

• The driver automatically gets shipped in all Linux distributionswithout having to ask the distros to add it.

As Linux supports a larger number of different devices “out of the box”than any other operating system, and it supports these devices on moredifferent processor architectures than any other operating system, thisproven type of development model must be doing something right :)

Thanks to Randy Dunlap, Andrew Morton, David Brownell, Hanna Linder,Robert Love, and Nishanth Aravamudan for their review and comments onearly drafts of this paper.