A Conversation with Kernel Developers from Intel, Red Hat and SUSE

Three kernel developers describe what it's really like to work on thekernel, how they interact with developers from other companies, some petpeeves and how to get started.

Like most Linux users, I rarely touch the actual code for the Linuxkernel. Sure, I've looked at it. I've even compiled the kernel myself on ahandful of occasions—sometimes to try out something new or simply tosay I could do it ("Linux From Scratch" is a bit of a right of passage).

But, unless you're one of the Linux kernel developers, odds are you justdon't get many opportunities to truly look "under the hood".

Likewise, I think for many Linux users (even the pro users, sysadmins anddevelopers), the wild world of kernel development is a bit of a mystery.Sure, we have the publicly available Linux Kernel Mailing List (LKML.org)that anyone is free to peruse for the latest features, discussions and(sometimes) shenanigans, but that gives only a glimpse at one aspectof being a kernel developer.

And, let's be honest, most of us simply don't have time to sift through thecountless pull requests (and resulting discussions of said pull requests)that flood the LKML on a daily basis.

With that in mind, I reached out to three kernel developers—each workingat some of the most prominent Linux contributing companies today—to askthem some basic questions that might provide a better idea of what being aLinux kernel developer is truly like: what their days look like and howthey work with kernel developers at other companies.

Those three developers (in no particular order):

• Dave Hansen, Principal Engineer, System Software Products at Intel.
• Josh Poimboeuf, Principal Software Engineer on Red Hat Enterprise Linux.
• Jeff Mahoney, Team Lead of Kernel Engineering at SUSE Labs.

Intel, Red Hat and SUSE—three of the top contributors of code to theLinux kernel. If anyone knows what it's like being a kernel developer,it's them.

I asked all three the exact same questions. Their answers are here,completely unmodified.

Bryan Lunduke: How long have you been working with the Linux kernel? What got youinto it?

Dave Hansen (Intel): My first experience for the Linux kernel was a tiny littledevice driver to drive the eight-character display on an IBM PS/2, probablyaround 20 years ago. I mentioned the project on my college resume, whicheventually led to a job with IBM's Linux Technology Center in 2001. IBM iswhere I started doing the Linux kernel professionally.

Josh Poimboeuf (Red Hat): My first introduction to Linux happened around 2001, whenI was a software engineer at IBM working on server firmware. IBM hadrecently embraced Linux, and I was placed on a team that was responsiblefor replacing legacy proprietary firmware with a new embedded PowerPCplatform based on Linux.

Once I discovered Linux, I was hooked. I installed it on my laptopimmediately. It was mind-boggling that all the source code was freelyavailable, and that you could control every single bit of code that ran onyour laptop. And, unbelievably, it was free.

At IBM, I started out by doing hardware bringup and Linux applicationdevelopment. But I was always especially fascinated by the kernel. So mycuriosity gradually led me to work my way down the stack. My first realkernel experience came when I started writing device drivers in 2004.

By 2008 I was the "kernel guy" on the team, responsible for porting thekernel to our proprietary HW, and for resolving all kernel issues found inthe field. That was a bit of a trial by fire, but it was a great way tolearn about the entire kernel tree.

The kernel is so big that my learning process still continues to this day.That's always been one of my favorite things about the kernel. There'salways more to learn.

These days I work at Red Hat, where I do a lot more upstream work. I workon a wide variety of things: live patching, the objtool static analysistool, the x86 unwinder, speculative CPU vulnerability mitigations andmore.

Jeff Mahoney (SUSE): I've been working with the Linux kernel for 20 years. I gotinto it initially in college because I was interested in systems software.I happened to buy some hardware for which there was no driver, and I wrote asmall one. Before working on the kernel, I was a systems admin for UNIXsystems, and a coworker and I then decided to try writing a clustered filesystem ourselves. It turned out that the hardware to do that was much tooexpensive for us, so we ended up contributing to ReiserFS instead. That ledboth of us into careers working on Linux.

Bryan: What does a regular day as a Linux kernel developer look like?

Dave (Intel): My days can vary a lot. The one constant isprobably email—lots and lots of email. It might be internal or external code reviews, oranswering a question from another Intel team or an external customer. Thebest, most satisfying days are the ones where you start with a problem or akernel crash, and have a patch posted by the end of the day.

Josh (Red Hat): It might actually be surprising to learn the variety ofthings a kernel developer does on a daily basis. Each day—and week andmonth—is different. It's often "choose your own adventure".

Obviously, one of the main things a kernel developer does is write code.Sometimes I can (mostly) disappear for a week or month (or two!) to hackaway on a new feature. Those days/weeks/months are my favorite part.

But writing code is only part of it. There's also debugging, reading code,collaborating, testing, code review, code-related discussions, readingpapers, research and meetings. It's good to mix things up. And you get tointeract with some really smart people from all over the world, which isreally interesting.

Most communication happens through email, but many kernel developers alsoattend conferences, like Linux Plumbers Conference or Kernel Recipes. Alot of good discussion happens at conferences. They're also useful forputting faces to names, which makes a big difference when you're mostlyinteracting with people over email.

Jeff (SUSE): It's a mix of communication, coding, building and testing.Lots of email—bug reports, code review, design discussions eitherinternally, on public mailing lists or IRC.

Bryan: With the kernel work your team does, how much of it is workingwith others within your own company, and how much is working withdevelopers working on Linux at other companies? Possibly companies thatcompete with each other in some ways?

Dave (Intel): Because of the incredible variety in the way that ourcustomers use Linux, work with the upstream kernel is an absoluteimperative. Virtually all of that work results in work with folks in thecommunity from other companies. There is also a lot of work that goes onbehind the scenes to support the work with the community. A great commentfrom someone in the community might result in us going off for a week or amonth to revise our work. Although we might not be sending mail on LKMLon a given day, we are actively working with the community.

Josh (Red Hat): It really varies by person. Some people spend 100% oftheir time working upstream with the Linux community. Others spend 100% oftheir time internally, backporting and resolving issues in RHEL. Many ofus are somewhere in the middle, dividing our time in both worlds.

Red Hat's policy is "upstream first". So any features or fixes in RedHat's kernel have to be accepted by the upstream community first. Thatgives us plenty of opportunities to work with the Linux community.

Jeff (SUSE): It depends on what we're doing—bug reports tend to behandled within SUSE, by our developers and support teams. Part of that iscontractual, while part of it is practical. When we release a product,we've chosen a particular kernel version and build on top of that. Anyfixes must also be against that version, and the upstream community isn'tusually interested in those. Once we've created a fix, if the bug stillexists in the latest release, we'll do that work in public.

Feature development happens on public mailing lists, where the participantsmay work at SUSE, may work for other companies, or may be doing it out ofpersonal interest. One of the most enjoyable parts of working on Linux isthat even though there are developers from hundreds of different companieswho may be competing with one another, we get to collaborate as if we wereon a single team. In addition to the mailing lists used to do code reviewand discussions, many subsystems have IRC channels where developers (andusers) chat about projects and socialize.

Bryan: When you need to work with other companies (be it Intel, SUSE, RedHat, Canonical, IBM and so on) on kernel issues—such as securityvulnerabilities—how does that work? Is there an established process?

Dave (Intel): There are really two complementary processes that happen.Intel has formal company-to-company communication channels that are reallygreat for synchronizing the business side of things. A challenge on thesecurity front has been creating communication channels that support thecoordinated disclosure process and simultaneously support normal communityprocesses, like mailing lists. Both avenues have matured quickly andcontinue to evolve to help us meet the evolving security landscape.

Josh (Red Hat): When there's an embargoed security vulnerability, we dohave strict processes in place for secure collaboration with othercompanies.

Luckily, such embargoes are rare, and they're the exception to the rule forhow we normally operate. We typically work closely with developers fromother companies all the time on the Linux kernel mailing list, with nospecial processes needed.

One good example is live kernel patching. My team at Red Hat created thekpatch technology, but at the same time, a team at SUSE created kGraft.Instead of going forward with competing approaches, we worked closely withthe SUSE team at conferences and through mailing lists to create livepatch,which actually turned out to be a better technology than both kpatch andkGraft.

In fact, cross-company collaborations like that happen every day on theupstream mailing lists. It's really just business as usual. Ourinteractions are always focused on what's best for upstream. In the end,what's best for upstream is also what's best for the companies that relyon it. That independent company-agnostic attitude is strongly reflected inthe upstream Linux culture.

Jeff (SUSE): For security vulnerabilities that aren't yet public, oursecurity team coordinates with their counterparts with other companies.Otherwise, unless there's a compelling reason not to, the collaboration allhappens on public mailing lists. There, the process is to post your code,listen and respond to review and feedback, perform the required changes,re-post, and repeat.

When the feedback is positive, the process is complete, the maintainer forthe subsystem will pick it up (according to their timeline) and pull itinto the git repository for their subsystem. Then the maintainer asks Linusto pull those changes into the mainline repository.

Bryan: Every software developer has a pet peeve with the projects theywork on. What's your pet peeve—the thing that you really wish you couldchange—with Linux?

Dave (Intel): I really wish developers would focus on making reviewers'lives easier. First, communicating what you are doing, why you are doing itand why it matters is critical. Then, making sure that the code and itssupporting comments are as self-explanatory as possible. Too often, wefocus on making sure the code functions, then call it a day. To me, that'sonly half of the job.

Josh (Red Hat): If I had a magic Linux wand, I would:

1. Eliminate CPU speculation—no more Spectre/Meltdown-type bugs!
2. Get rid of the need for security embargoes—but to be clear, I believethat such embargoes are necessary in the real world.
3. More broadly diversify the Linux kernel development population. Morediffering perspectives can produce better ideas. I think we're alreadyslowly moving in that direction.

Of those, #1 and #2 aren't realistic, but maybe we can achieve #3.

Jeff (SUSE): The lack of diversity in the community, especially the gendergap. Women are underrepresented in computer science fields generally, butespecially so in the Linux kernel community. There has been some outreachefforts, but more needs to be done.

Bryan: The Linux kernel is, at this point, more than a quarter of a centuryold. In software terms, it's certainly been around a while! Do you see theneed for it to be replaced any time soon? If so, with what? Ifnot, why?

Dave (Intel): I don't really see Linux as a 25-year-old project. TheLinux of 25 years ago was not the de facto OS on servers, routers orphones. There's no need to replace something that's continually changing,growing and improving as fast as Linux.

Josh (Red Hat): These days, tech trends are fickle, and most technologieshave a very short lifetime. But I don't see Linux going anywhere. Itstrue strength is in its development model. It's not perfect, but it'sstill the best way to produce software at scale that I've ever seen. Iwouldn't be surprised to see Linux thrive well into the next quartercentury.

Jeff (SUSE): While the project is more than 25 years old, it hasn't stoodstill. The kernel itself is constantly evolving to meet new needs. Newkinds of users are coming to the Linux community every year. The kernel hashad between 10,000 and 15,000 commits in each release for at leastthe past ten years. The community is still growing. I don't think it willbe replaced any time soon, but it will continue to evolve.

Bryan: What would you tell folks thinking of getting into Linux kerneldevelopment?

Dave (Intel): Please do! Linux is only becoming more important tocompanies like Intel. It's a challenge to find folks with the technicalskills to work on the kernel and the skills necessary to navigate thecommunity. The most successful folks who join the community are the onesthat have aproblem to work on. Maybe it's some device that Linuxdoesn't support, or a bug that's driving you crazy on your laptop. Thefolks that come with patches that don't solve a clear problem generallyhave a tough time getting those patches accepted.

Josh (Red Hat): First, I'd say to just find a way to dive in and see ifyou like it.

One good way to get started is to pick a small area of the kernel you'reinterested in and dedicate yourself to becoming an expert on that littlepiece of code. Read the code until you understand it. Tweak it and seehow it affects your system. Start reviewing patches related to it on themailing list. After a while, you'll start seeing opportunities forpatches, like bug fixes or code improvements.

When you do eventually post a patch, don't get overly attached to yourcode. Try not to take feedback personally. Our common goal is to producethe best code. It's ok to make mistakes. Put your ego aside, be humble,be respectful, and listen to feedback with an open mind and try to learnfrom it. That's how the code gets better. It also helps you earn respectfrom others in the community.

Kernel development can take a lot of patience, humility and persistence.It's not uncommon for code to be thrown away or rewritten several times.The process can seem inefficient at times. But in my experience, the endresult is always better than anything proprietary development can produce.

The kernel codebase is huge, so diving into code you've never seen will bea common occurrence. Whenever you have a question about how somethingworks, the answer is always in the code somewhere. Get familiar withcscope. For vim users, I'd recommend the vim cscope plugin.

Also, work on your written communication skills, as most of your non-codingtime will be spent in email. And, of course, learning to make your codeeasily readable by others is also very important.

Finally, finding a mentor (or mentors) can be valuable. I never had anofficial mentor per se, but I've been lucky enough to have had many moreexperienced people guide me through the years.

Jeff (SUSE): It can be a lot of fun, but it takes some effort to get started.Start with something you're interested in, find something small to fix, andpost your work. Read about and understand the process. Listen and respondto feedback. Experienced developers are usually willing to spend some timehelping new developers if they're willing to listen.

Bryan Lunduke is a former Software Tester, former Programmer, former VP of Technology, former Linux Marketing Guy (tm), former openSUSE Board Member... and current Deputy Editor ofLinux Journal, Marketing Director for Purism, as well as host of the popularLunduke Show. More details:http://lunduke.com.